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ABSTRACT 

This study began with the pc vchologi cal postulate that all 
human performance in a choice situation tends to be made on a 
systematic basis. 

In the setting of multiple choice achievement tests, this 
postulate resolved itself into three operational hypotheses which form 
a necessary and sufficient set to establish the possibility that it 
applied to both the right and the wrong answers given by examinees. 
Testing these three hypotheses involved the tallowing procedures: 

1) To develop and logically validate a systematic method for 
the construction of the foils (di s tractors) on a multiple 
choice achievement test designed tc measure higher mental 
processes • 

2) To show that the construct validity of this systematic 
method held up reasonably well in the results of the 
administration of this test. 

3) To show that the validly produced foils in this context 
improved the predictive validity of this test with respect 
to other achievement tests over the more usual procedures. 

The results of the study tended in general to support these 
three hypotheses fairly strongly if we take into account the finding 
that many of the foils could be classified into more than one category 
as evident in the low interrater reliability, the need to reclassify 
foils v/ hen wrong answer patterns were being interpreted, and the manner 
in which these interpreted foil categories cross-validated. This study 
would seem to have produced three fairly definite findings: 


to multiple 


1. Human perforttiiance , wags. abstracted from responses 

choice achievement tee to involving higher mental processes, 
would seem to bo systematic, and to display evidence of 
multiple interpretation of the conununi cation <> 

2c. There would seem to be a hierarchy of foils which parallels 
the hierarchy of right answers and which influences the way 
in. which each total i tern v v?rf oms <• The levels of the foils 
themselves seem to depend upon the ways in which this 
totality of each item is approached 0 
3o Wrong answers contain potentially useful information with 
respect to achievement when higher mental processes are 
involved o 

Taxonomic tests v/ould seem to have a rvchbor of properties not 
assumed to be present v/hen the lest is sufficiently homogeneous to be 
assumed to form a scale* The existence of these properties made it 
fairly evident that the more commonly used analytic procedures were 
probably inappropriate for the analysis and interpretation of the results 
from, and the criteria for* evaluating the effectiveness of this type of 
test o The alternative analytic procedure which the findings of the study 
implied v/ere organized into a suggestion for the extension of test theory- 
designed to deal with the problems which seem to arise where taxonomic 
tests are concerned 0 

Some implications of the findings to educational practice were 
drawn, and a number of suggestions for future research into this area 
were presented 0 
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AB3TKAC? 


This study began with the psychological postulate that all 
human performance in a choice situation tends to be made on a 
systematic basis. 

In the setting of multiple choice achievement tests, this 
postulate resolved itself into three operational hypotheses which form 
a necessary and sufficient set to establish the possibility that it 
applied to both the right and the wrong answers given by examinees. 
Testing these three hypotheses involved the following procedures: 

1) To develop and logically validate a systematic method for 
the construction of the foils (dis tractors) on a multiple 
choice achievement test designed to measure higher mental 
processes . 

2) To show that the construct validity of this systematic 
method held up reasonably well in the results of the 
administration of this test. 

3) To show that the validly produced foils in this context 
improved the predictive validity of this test with respect 
to other achievement tests over the more usual procedures. 

The results of the study tended in general to support these 
three hypotheses fairly strongly if we take into account the finding 
that many of the foils could be classified into more than one category 
as evident in the low interrater reliability, the need to reclassify 
foils when wrong answer patterns were being interpreted, and the manner 
in which these interpreted foil categories cross-validated. This study 
would seem to have produced three fairly definite findings: 


1. Human performance, when abstracted from responses to multiple 
choice achievement tests involving higher mental processes, 
would seem to be systematic, and to display evidence of 
multiple interpretation of the conununi cation 0 
2o There would seem to be a hierarchy of foils which parallels 
the hierarchy of right answers and which influences the way 
i -n which pp.ch tot a*’ item ^orf oms < The levels of the foils 
themselves seem to depend upon the ways in which this 
totality of each item is approached 0 
3o Wrong answers contain potentially useful information with 
respect to achievement when higher mental processes are 
involved o 

Taxonomic tests would seem to have a number of properties not 
assumed to be present when the test is sufficiently homogeneous to be 
assumed to form a scale. The existence of these properties made it 
fairly evident that the more commonly used analytic procedures were 
probably inappropriate for the analysis and interpretation of the results 
from, and the criteria for, evaluating the effectiveness of this type of 
testo The alternative analytic procedure which the findings of the study 
implied were organized into a suggestion for the extension of test theory 
designed to deal with the problems which seem to arise where taxonomic 
tests are concerned 0 

Seme implications of the findings to educational practice were 
drawn, and a number cf suggestions for future research into this area 


were presented 
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CHAPTER I 


11TPR0DITCT ION 

Psycho] ogioal literature has been replete with studies of choice 
behavior, Cho wn (1959) 9 Duncan (1959) and more recently Hunt ( 1961 ) and 
Berlyne ( 1965 ) have reviewed this literature adequately 0 From these 
studies it is fairly evident that the distribution of choices made by 
humans in problem-solving situations tends to exhibit some systematic 
trends . 

These trends, however, have often been confined to discussion in 
terms of the patterns of "success" when compared with the nature and 
complexity of the task. Many studies, for example Strutz ( 1966 ) con- 
centrate mainly on ’’right" answers and the relevant patterns involved. 

A notable second direction in this area has been the attempts to 
describe the nature of the procedure used by the person in his attempt 
to solve problems; for instance, Piaget (1953) L proposed a logical 
system for these procedures. Abel son and Rosenberg (1958) proposed a 
logical system which they call "psychologic" which proposes a set of 
"logical" procedures which lead to certain types of "wrong" answers. 

Specifically, within the area of achievement testing the 
multiple choice test provides a good opportunity to observe choice 
behavior in a problem-solving setting. With the advent of Bloom’s 
Taxonomy (1956) a considerable improvement in the classification of 
test items involving these "higher mental processes" became possible 0 

This present study will be to l) demonstrate the presence or 


1 Furth ( 1969 ) discusses the Piaget model in some detail. 
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absence of this additional information. 2) identify the general 
properties, if any, of this information, and 3) speculate as to the 
implications of such findings 0 Assuming systematic choice behavior, 
consistencies would be expected within and between individuals for all 
choices made. These consistencies may not be confined to the 
M successes n or "right 11 answers. This study proposes to explore 
certain aspects of the possibility that the choices made among wrong 
answers may also be systematic and therefore contain useful information 
for the examiner c 

Statement of the_ Problem 

Some aspects of the possibility that wrong answers are selected 
systematically have been examined /(?f Fouldes said Forbes, 1965$ Powell. 
1968; Jacobs and Yandeventer , 1968; Powell and Isbj.ster, 1969/, and these 
studies are discussed in more detail in Chapter II Q 

The specific concern of this study is to show to the designers 
of tests the significance of wrong-answer information© Any significant 
improvement in a test must be reflected in a corresponding improvement 
in the validity of the test. This study proposes to examine the 
construct and predictive validities of a particular method of test 
construction o The purpose of this study, then, is to explore the 
possibility that, if tests are constiucted in a particular manner, 

"wrong" answers may add to the examiner* s information about the 
examinee. This present study will be content to demonstrate the 
presence or absence of this additional information to determine the 
major properties of this information, and to speculate as to the 
implications of such findings. 


CHAPTER II 


BACKGROUND FOR THE STUDY 

As already mentioned in Chapter I, this study was essentially 

exploratory. For this reason, little attempt has been made to 

establish a theoretical rationale upon which the hypothetical structure 

of the study might be built. Instead, the study was organized on the 

basis of procedural considerations. The absence of a theoretical 

rationale for the interpretation of results had the advantage that the 

data could be examined for consistent characteristics and the 
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properties of these characteristics. 

Of course the nature of the procedures employed provide 
definite limitations upon the interpretations. The findings themselves 
would tend to provide other limitations upon the interpretation, and 
also the generalizability of the findings. 

It is the purpose of the present chapter to review the most 
significant research which is relevant to the present study in order to 
present the research background which forms the basis for the 
procedural considerations which were employed. 

The Problem of Measuring Academic Performance 

The functions of, and therefore the outcomes of, education are 
a subject of debate which is beyond the scope of this dissertation., 
However, these functions and their corresponding outcomes have an 
important bearing on the nature of and the interpretations given to 
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Glaser and Strauss (196?) developed a complete rationale to 
justify this procedure for exploratory studies. 
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the results of the various kinds of measuring instruments used 0 

Tests are formalized communications between examiner and 
examinee o The examiner is attempting to obtain a controlled sample of 
behavior to assist him in the rendering of certain judgments 0 These 
judgments may be either value judgments (continue, withdraw, certificate); 
or they may be procedural (concerning the nature of the appropriate treat- 
ments of programmes) * The greatest possible control of the behavior 
sample is found in the multiple choice test. 

As communications , tests involve several important considerations 0 
First, they involve the examiner 1 s perception of the examinees in terms 
of the capabilities they do have, and the capabilities they should have 
(educational goals ) 0 These perceptions lead to the examiner’s decisions 
as to which information to give in the examination, in what format, and 
which information to withhold. 

Second, on the basis of these considerations a communication is 
formulated. For the purpose of this study these communications will be 
confined to the multiple choice achievement test D 

Third, the communication is presented to the examinee who is 
expected to interpret it and to respond to it 0 He will do so on the 
basis of the capabilities he possesses; the information he possesses, 
particularly that part of the information pool which was withheld by the 
examiner; and his sense of the importance to him, as a person, of the 
answers he gives including the information about himself which he wishes 
to withholdo 

Fourth, the examiner then has the task of interpreting the 
responses of the examinees and making such value or procedural judg- 
ments as may be appropriate to these interpretations and to the 
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purpose of the test. To form these judgments the- examinee's performance 
can be compared, as appropriate, with l) his own past performance in 
similar contexts, 2) the performance of others in the same context 
(norm referencing) or 3 ) some external behavioral definition of mastery 
(criterion referencing) . 

However, where the subject matter content is itself open to 
disagreement, examiners themselves may not agree as to the appropriate- 
ness of the communication or its interpretation 0 Also, the examiner's 
assumptions about the capabilities and information background of the 
examinees may not be congruent to their actual characteristics. Further- 
more, there may be little similarity between the examiner's purposes 
and the examinee's interpretation of these purposes 0 In addition, if an 
examinee has systematically misclassified a particular concept and this 
concept recurs with a. high degree of frequency in a test, the examinee 
is likely to obtain a low total-correct score. Given the opportunity to 
correct this misclassif ication could lead to a much higher total score 0 
How serious, then, must a misclassification be considered? Finally, 
suppose that the examiner misclassifies? Such an event is bound to have 
an adverse effect on the total correct score of the profoundly informed 
student as Hoffman (1962) points out. The combined effects of such 
considerations upon the composition of total-correct scores complicates 
their interpretation. 

Technical Consideration in the Measurement of Choice Behavior 

This study will confine itself to the choice behavior of 
examinees as exemplified in their responses to multiple choice achieve- 


ment tests. The particular point of view to be expressed is relative to 
the way in which current practice tends to use wrong-answer informant i on 0 
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Current pract /i cc for scorl ru achievement tests r Present 
practice for scoring’ multiple choice achievement tests is to count the 
number of "right M answers selected by the examinee on a test or a 
subtest . The "right" answers are usually predetermined although 
experience with particular items may lead to subsequent revisions. 

In such tests the examinee is faced with several alternatives only one 
of which is "right." This means that he can make a wrong choice among 
several alternatives 0 In general, however, distinctions which might be 
made -among students on the basis of differences among the wrong answers 
selected are nor considered when the students 1 scores are evaluated 0 
If wrong answers are used for any purpose it is usually to correct the 
scores for guessingo 

Current Practice for Evaluating A chiev e ment Tests - - Reliability 

There are three general areas in which the specific 
characteristics of a test can be improved 0 These are: 

1. Reliability 

2. Validity 

3o Useability 

The third characteristic of these, useability, can be dispensed 
with quickly because the simplicity of administration, and the 
simplicity and objectivity of scoring of multiple choice tests and the 
ease in the establishment and use of norms has been well established,, 
The other two characteristics require more discussion 0 

The reliab i lity of a test . The concept of test reliability 
involves how well the test measures whatever it measures. The APA 
Standards ( 19 66 ) lists three methods of estimating reliability. 
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These U 2 'G ; 

1 . In t e 1*1 i a. 1 c on s i s t en oy 

2. Reliability between forms 

3. Reliability over time 

The latter two are determined by correlation coefficients either 
between alternative forms, or between repeated administrations of the 
same form on the same group of examinees. Neither of these two 
approaches is directly applicable to this study. 

There are several possible approaches to the study of internal 
consistency. These are: 

1. Item analysis 

2. Using a correction-for-guessing 

3. Designing the test to form a scale 

4. Examining internal characteristics of the test 

5. Part- whole comparisons 

Internal consistency from i tern analysis . There are two schools 
of thought with respect to item analysis. The classical approach 
assumes that all of the items on the test should measure the same over- 
all characteristic that the whole test, or that the relevant subtest, 
measures. To determine this similarity, the distribution of right 
answers on a particular item is correlated with the distribution of 
total- correct scores. The biserial correlation coefficient is usually 
used. This correlation coefficient actually correlates a dichotomous 
variable with a continuous variable. For the answers of a multiple 
choice item to bo dichotomous, the plural set of wrong answers must be 
treated as a single variable. Similarly, the discrete distribution of 


8 


total scores ^ the xotal scores a 2 *e the sum of' a binary vector ol‘ "right - 
. wrong” decisions which sum must be a whole number) must be treated as a 
continuous variable. The problem that score data provide discrete 
distribution is avoided by assuming that total scores are "best 
estimates” of ”true scores” and true scores are assumed to form a 
continuous distribution. There is, of course, a multiserial correlation 
/Cf Jaspen, l^\bj which could be used to take account of the plurality 
of "wrong” answers. This latter coefficient is rarely used to evaluate 
multiple choice test items. Classical test theory advocates that the 
biserial correlation coefficient for each item should be high 
(signif icantly different from zero). 


An alternative approach suggested by Lord (1952) involves the 
considerations necessary for addition of scores. In order to add two 
numbers, they must be independent, that is, the set of lattice points 
each represents can share no elements in common. By this approach, 
individual items should be relatively uncorrelated, but should 
collectively form a scale. 

Both of these procedures tend to treat a multiple choice item 
as a dichotomy thus overlooking the fact that more than one choice can 
be made among the set of foils. 


Using a correct! on- for- gue s sing ,, Originally, guessing 
corrections for scoring formulae assumed that the number of right 
answers which are attributable to guessing were directly related to the 
number of wrong answers given and inversely related to the number of 
alternatives per item. Because any answer could be a "guess,” no 
meaning could be ascribed to particular answers. 


Meaning was thus 
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assumed to be confined to some form of cumulative score, . This 
correction has the effect of increasing the variance of the total scores 
because a greater amount .is subtracted from the low scores than from the 
high ones. ‘ 


With respect to corrections-f or-guessing, Gupta and Penfold 
( 1961 ) shev/ed that the guessing correction over-eorrects in the event 
that the examinee is responding on the basis of misinformation. A 
similar argument can be presented to suggest that this correction uncer- 
corrects the partially- informed examinee. More recently, Shufora, and 
Massengill (1965) elaborated upon a system of "confidence scoring” in 
which the examinee rates every alternative on the basis of his con- 
fidence that each particular alternative is right. Honesty is 
encouraged on the basis that "confidently wrong" loses marks. This 
procedure makes it possible tc classify each examinee’s answer to each, 
question as l) well informed, 2 ) partially informed, 3 ) uninformed, 
and 4 ) misinformed. This procedure solves the guessing correction 
problem by identifying which items were "guessed" thus increasing the 
interpretabi.lity of particular items and hence the validity of the test. 
The scoring method these authors developed increases the internal con- 
sistency of the test by increasing the true score variance estimates 
proportionally more than the total score variance. 

Design: rig a t e s t to form a scale . The argument may be raised 
that the practice of distinguishing among individuals on the basis cf 
total scores without considering the constituents of those scores may 
produce information loss. This argument can lead to the proliferation 
of subtests, or it can lead to test designs in which the scores form a 
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scale. For instance, Cox and Graham ( N 1 966) propose a system for 
designing a test which uses Gagne's task analysis /CS Gagne , 15 * 65 , 
Chapter VI l 7 , to produce a Guttmann ( 1954 ) scale. In this case, the 
score may indicate the level of mastery. Again, the internal 


consistency of the test can be increased, this time by increasing 
item homogeneity. 


Examining internal best characteristi cs . One other area which 
has led to improvements in the reliability of tests has been through 
research into the improvement of the definition of the variables being 
measured by a test. Research toward this objective has been more 
extensive in the area of personality tests than in the development of 
achievement tests. The design of personality tests is beyond the scope 
of this present study. In view of the scarcity of appropriate research 
from the achievement testing area, only two developments in this latter 
testing area will be discussed here. First, Ayers (1965) attempted to 
validate Bloom's Taxonomy by means of factor analysis from tetrachoric 
correlations using programmed instruction in order to control the 
teacher variable. His findings in general supported Bloom's notion of 
a hierarchical structure. However, the results did not consistently 
fit the classification system in the Taxonomy . A more ambitious study 
to this same end was conducted by Kropp, Stoker, and Bashaw (1966). 
Although their findings were similar to those of Ayers (1965) because 
of the illumination their study provides for the construction of 
taxonomic tests, it is discussed in detail on page 18 ff. For our 
present purposes, it may be sufficient to say that the validity and 


the reliability of achievement tests may be improved by using Bloom's 
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T axonomy as a gu .i de for developing the .items. 

Second, Gupta (1968) snowed that the reliability in an internal 
consistency sense of an achievement test can be improved if the test is 
subdivided into subtests based on factor analytic results or on the 
basis of the DuBois, Loevinger and Gleser (1952) method of cluster 
analysis. This procedure makes subtests from relatively homogeneous 
items. This present study used a similar approach. 

It should be noted, once again, that these methods tend to 
concentrate exclusively on the "right M answers. 

Reliabil: t f y based on part - whol e comparisons . A special case 
of the alternative forms method of determining the reliability of the 
test is the group of procedures which use the correlation of one part 
of the test with another. The mathematical limit of the repeated use 
of the split-half technique when certain assumptions are made is found 
in the Kuder-Ri chardson (K-R) formulae. It is this form of reliability 
which increases in the DuBois et al (1952) procedure. 

The Kuder- Richardson procedure is most sensitive to differences 
in the variance of the test. For this reason, if error variance is 
kept constant, increasing the test variance (as when using a 
correction-f or-guessing) , also increases the reliability. Another 
method of increasing the variance is to rev/rite the test in such a man- 
ner as to move the difficulty (selection ratio) of each item toward .5 
(50 fo) . If the item is a dichotomy, a difficulty of .5 (50 %) tended to 
maximize the variance assuming positive correlation because it 
maximizes the probability of choosing either alternative. 

It is a common suggestion in evaluation texts for example, that 
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items should have middle range of difficulty. This suggestion assumes 
that each item should be treated as a "right. -wrong" dichotomy. The 
plurality of wrong answers is being overlooked when items are treated 
as a dichotomy. 

Current Practices for Evaluating Achievement T ests — Valid! ty 

In addition to the reliability of a test it is also necessary 
to be sure that a test measures the things it is intended to measure, 

i.e. the validity of the test. The validity and the reliability are 
related in that the validity of a test can never be higher than the 
square root of the test's reliability v/hen the latter is defined in 
repeated-measures terms, hence the efforts to increase test reliability. 

The APA Standards lists three types of validity. These are: 

1. Content validity 

2. Construct validity 

3 . Criterion-related validity 

The concept of content validity refers to the validity of an 
item or test as dependent upon the appropriateness of the item or test 
of the information background needed to answer the test. In this 
present study most of the necessary information background needed is 
supplied in reading selections embedded in the test. 

The construct validi ty aspect of a test. Construct validity 
has several aspects. In brief, construct validity refers to some 
psychological construct or constructs, more or less independent of 
content, which are included in the test. For instance, if intelligence 
as a construct is assumed to he manifested by intelligence measures, 
such as the VISC, the correlations between a new group I . Q 0 test and 
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the WISC scores on the same subjects could be support for the construct 
validity of the new test. In this case the construct would be 
'’intelligence . ” Another approach to construct validity is to define 
the construct in such terms as to facilitate translation into perform- 
ance terms. A good example to this approach is Bloom’s Taxonomy (1956). 
As already indicated, this procedure should increase the reliability of 
the test as well as its validity. These constructs should also be 
identifiable in the performance of examinees when the performance data 
are subjected to statistical analysis • 

Another procedure which strengthens support for constructs for 
which measures have not been standardized is ’’cross-validation.” In 
cross-validation statistical analysis should reveal the same constructs 
in independent groups. Cross-validation is used in this study for 
testing the construct validity of the procedure being explored here for 
the development of foils (wrong alternatives) on multiple choice tests. 

A final aspect of construct validity concerns the degree to 
which the examiner's objectives have been accomplished by the test he 
has developed. In the absence of standards this accomplishment is 
difficult to measure. One approach is to study the distribution of 
answers to an item to get clues to its effectiveness. Fart of the 
discussion in Chapter III on the development of the experimental test 
used in this study will elaborate this procedure. 

In some cases the construct may be sufficiently well defined 
that the different performance outcomes are indisputable. In such 
cases the construct validity of a test may be easily determined. 

Piaget’s discussion of the acquisition of various aspects of 
conservation concepts are a case in point. Items measuring the 
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acquisition of these concepts must conform in their discrimination to 
the known characteristics of this acquisition process. Where wrong* 
answers are concerned, as wil'J be discussed on page 27 ff no such clear 
definition exists. The present study, therefore, can be no more than 
exploratory in nature. 

Criterion-or i ented aspects, of a test . One of the fundamental 
functions of any measurement of achievement is its predictive value for 
future achievement. Within the context of the present study one of the 
concerns is the ability of the experimental test which may, by its 
construct characteristics, he considered as a test of strategies which 
may improve the prediction of other achievement test results. Popham 
and Husek (19&9) point out that most of the statistical procedures used 
in current practice may be inappropriate for criterion-referenced tests. 

Studies Related to Wron g Answers 

At this point the concept of answering patterns becomes 
critical. An answering pattern will, for present purposes, be defined 
as some characteristic among tlqe answers selected by a group of 
students which is consistent and stable under statistical analysis, and 
hence leads bo an improvement in the validity and reliability estimates 
of the test to which these answers are given. The works already quoted 
suggest that there may be such patterns among '’success” performance. 

The question can now be raised as to whether or not there may be 
answering patterns among wrong answers as well. 

A possible source of findings .concerning wrong answer informa- 
tion is diagnostic testing which has a considerable history. Schonnel 
(1943) discusses the cumulative results of more than twenty years of 
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research. His procedures snowed that the nature and location of 
mistakes can reveal specific problems, i.e. wrong answers can be meaning- 
ful for diagnostic purposes.. However, error types are usually estab- 
lished in advance, and are restricted to items which reflect only one 
type of error thus allowing the items to be scored, as right or wrong*, 
Large numbers of questions are needed for this procedure since as the 
complexity of the problem to be solved increases, the number of tasks 
required to diagnose all possible errors increases expotentially; prob- 
ably explaining the absence of diagnostic tests in '’subjective 1 ' subjects. 
The Cox and Graham (19 66) procedure refines this diagnostic technique 0 
In order to develop diagnostic tests of this sort, an interlocking 
pattern of items is usually designed, in such a way that specific weak- 
nesses in a particular students performance can be inferred*. This pro- 
cedure identifies weaknesses on the basis of relationships between items 
rather than relationships between alternatives within a particular item. 

It becomes evident from the fact that diagnosis can lead to the 
identification of specific error types that the four categories of 
students 1 responses made by Shuford £t al (1965) may be an over- 
simplification. Furthermore, it would seem reasonable that more than 
one error type could be accommodated in one item if a multiple choice 
forma,t we re used. In this latter case, there should be evidence of 
answering patterns among wrong answers. 

Answering patterns in foil selection . The evidence supporting 
the possibility that there may be answering patterns in the wrong 
answers as well as the right ones is sparse 0 Sigel (1963) reported 
with reference to intelligence testing that children tend to "be 
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consistent witnin themselves in the errors they make /p. b"ij • " 

Fouldes and Forbes (1965) reported in the manual for their 
revision of the Advanced Set of Raven ! s Progressive Matrices the 
following finding concerning common errors: 


Four types of common errors could he identified 0 
(a) Incomplete solutions,. There were errors due to people 
failing to grasp all the variables determining the nature 
of the correct figure required to complete a test item. 
Instead they chose a figure which was right as far as it 
went but was only partly correct... (.B) Arbitrary lines 
of reasoning. Here the figure chosen suggests that the 
person has used a principle of reasoning qualitatively 
different from that demanded by the problem... (c) 

Overdet ermine d choices. These were errors involving 
failure to discriminate irrelevant qualities in the figure 
chosen, o. (i)) Repetitions. These are errors made by 
people who simply selected a figure identical with one of 
the three figures in the matrix immediately adjacent to the 
space to be filled, /p. 20 7 

Fouldes and Forbes (1965) did not attempt to show whether or 


not these common errors were more characteristic cf 


s ome ind i v i duals 


than of others. 

These types of error would seem to be more related to some form 
of answering procedure based on the relational characteristics of the 
alternatives rather than on their informational characteristics. 

Powell (1968) fact or analysed some wrong answers derived from 
an administration of Gorham’s Proverbs Test (1956)« This test would 
probably be classified as a comprehension test by Bloom’s Taxonomy . A 
wrong answer pattern of four factors resulted. These were: 

1. Reduction of information to affect a simplification of the 
statement 

2. Addition of irrelevant information 

3. Substitution of elements 
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Replacement of proverb by one largely unrelated 
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If this list .is compared with the one by Fouldes and Forbes on 
page 16. we find, at least by description, Factor 1 remarkably like 
their Class C ( (Verde term med choices). Possible relationships between 
the remainder are less certain although Factor 4 and Class B may be 
related. Their Class I) is unlike Factor 3 but is very much like the 
"Word-Word Links" class present in the experimental test used in this 
study. A definition of this class is on page 38. 

However, Si gel (1963) went on to report that there "seemed to 
be no relationship between type of error and total score." /j> . 53 J In 
contradiction to Sigel, Jacobs and Vandeventer (1968) showed that 
within the context of Raven's Coloured Prcgres sive Matri ces and by 
using the Guttrnan and Schlesinger ( 1967 ) facet design, that a relation- 
ship often does exist between right and wrong answers. Ebel ( 1969 ) has 
shown similar systematic characteristics among True-False items. 
Furthermore, Powell and Isbister ( 1969 ) showed that some wrong answers 
can be related to right answers so as to adversely affect the high 
scoring students 0 The type of foil involved was the "irrelevancy." A 
definition of this class is on page 37 * An inconclusive trend in this 
same direction was found in Factor 4, page 17* 

Thus, the available evidence, scanty though it is, suggests 
that neither "misinformation" nor "no information" (leading to a hap- 
hazard answer) are sufficient to account for all the wrong answers 
given to multiple choice achievement tests, and that in certain 
circumstances specific foils may influence the total-correct score. 

If wrong answers contain achievement information, then the 
wrong answers which display systematic characteristics which acceptably 
support the construct characteristics of the experimental test should 
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improve the prediction of independent achievement scores for the same 
examinees • 

This improvement should occur in comparison with the prediction 
made by either the total-correct scores on the experimental test or 
some reasonable subdivision of these scores into subtest scores where 
the subtests also fit the construct characteristics of the test. 

Studies Related to I tern Gene ra t ion 

Perhaps the most ambitious attempt to develop tests reflecting 
Bloom ! s Taxonom y (1956) was the work of Kropp, Stoker, and Bashaw 
(1966). These researchers encountered a number of problems in their 
work some of which are discussed here along with the alternative 
procedures used in the design of the experimental test used in this 
study. 

The problems they encountered which are relevant to this study 

are : 

1. Problems arising from the "Knowledge” category of the 
Taxonomy 

2. The generation of Synthesis and Evaluation category items 
in multiple choice format 

3. Item analysis problems 

4. Problems arising from implicit assumptions in their study 
Although the fourth of these problems is probably the most 

important for present purposes, the discussion which follows considers 
each problem in the order given here. 

Problems aris mg from the "Knowledge" category of the Taxonomy . 
Krop p et al (i 960 ) spent some time discussing whether the "Knowledge" 
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category in Bloom* a Taxonomy is a legitimate category and, if so, what 
psychological processes other than recall this category might represent 
They .further compound the problem by basing their questions on reading 
selections supplied in the test* Thus, the legitimate question is 
raised as to the meaning of less than a perfect score on “Knowledge M 
items when all the information necessary for the answering of these 
items is contained in the reading selection used*, 

These researchers do not comment on the possibility that 
“Knowledge” items presented in an “open book” format may not be 
“Knowledge” questions in the sense of Bloom's Taxon omy at all. Instead 
these questions, in order not to be obvious, produce a test of search 
skills more commonly known in the literature on reading skills as 
“reading for details” /Cf Gray, I960, p® ll/. It is not surprising, 
therefore, that an important contributor to the “Knowledge” category in 
two of the grade levels is an unidentified factor consisting in grade 
nine of “Word Arrangements , Letter S e ts. and Symbol Production /p* 1 3l/ 
and in grade twelve of “Thing Categories, Locations, and Gestalt 
Transformations” /p # 13 « All three of these tests we re positively 
loaded on the unidentified factor for grade nine, and the “Locations” 
test is positively loaded on the unidentified factor for grade twelve. 
These unidentified factors add credence to the suggestion that the 
“Knowledge” category for the Kropp £t al (1966) tests may well be more 
related to search-skills than to recall. There is probably no logical 
method of testing “Knowledge” as defined by Bloom's Taxonomy when the 

3 

'These names refer to names of specific tests from the Kit of 
Reference Test s.-. (French, Ekstrom, and Price, 1963) which 
purport to define particular cognitive aptitudes*. 
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information bfu;k ground is supplied by the test, 
experimental test in this study, no "Knowledge 11 
generated . 


In the case of the 
category items were 


General :i ng .multiple c ho i ce synthes is and evaluation 1 terns . 
Another point made by Kropp et al (1966) was the difficulty of 
generating multiple choice items of the Synthesis and Evaluation 
Categories. One of the problems encountered in this respect is the 
restriction of a specific category in Bloom's Taxonomy for inductive 
reasoning to one subcategory of the Synthesis Category. Another sub- 
category adds "unique communication" requirements which are impossible 
to meet in a multiple choice format. The third subcategory involves 
producing a plan or proposed set of operations. Again, the open- 
endedness of this requirement restricts its employment in the multiple 
choice format. 

Second, if the "Synthesis" category is restricted to induction, 
the problem remains that the internal structure of a single reading 
selection is usually highly organized. For this reason, the generation 
of a large number of items which require an inductive combination from 
some components of this selection or an inductive generalization from 
these components is very difficult because both of these possibilities 
are either explicit or closely implicit in the passage. However, if 
more than one reading selection is included in a test of this type, it 
would seem to be a relatively simple matter to generate items which 
require inductive combinations between selections or inductive 
generalizations between selections 0 This latter procedure is used in 
the experimental test in this study. 


It is possible that the nature of the strategies employed by 
examinees when solving problems has an effect, on the effective classifi- 
cation of the item by the Tax onomy. Two outcomes would be expected in 
this case 0 First, the more familiar an examinee is with the content of 
the problem the lower the effective classification of that problem.. Sec 
end, the nature of the strategy shifts employed for generating foils may 
influence the strategies which the examinee ha.s to employ to answer the 
problem, which in turn may also affect the classification of the problem 
For instance, an item classified as synthetic on the basis of the stem 
alone or the stem and right- answer may become a comprehension item if th 
foils stress reading comprehension e Perhaps the rather surprising- 
apparent dislocations of the Evaluation items in the Kropp et al (19 66) 
study reflect this problem. It should be noted that the Evaluation 
category occurs in the second, third, and sixth positions in the ordered 
Simplex (Outtman, 195*0 analysis and additionally in the fifth position 
by mean score /c.f Kropp et al, pp 0 83? 8 7? 887o Greater elaboration of 
this latter problem occurs when the implicit assumptions of the Kropp 
et al (1966) study are being discussedo Only three Evaluation items are 
used in the experimental test because of its length (30 items). 


Problems of item analysis on taxonomic tests . Another problem 
which Kropp et al ( 1966 ) discuss at some length is the problem of item 
analysis for tests designed to measure levels in a Taxonomy 0 The 
Taxonomy was developed on the basis of the assumption that each higher 
level subsumes all lower levels and adds some unique characteristics of 
its own. Thus, as the level of the Taxonomy increases so does the 


complexity of the problems which are appropriate to this level 0 It 
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would therefore he expected that the difficulty of Items designed for 
each category would increase as the level of the Taxonomy for which 
these items were designed increased. Thus, the selection of items on 
the basis of approximate middle difficulty at each level of the Taxonomy 
in order to maximize discrimination would seem to be inappropriate. 

This subsumption property also implies that if any item were missed at 
any level of the Taxonomy , all items designed for higher levels of the 
Taxonomy which involve the context of the item missed should also be 
missed. As a result, the number of items correct at any level of the 
Taxonomy should determine the upper limit of the possible score for the 
next higher level. 

Kropp et al did not test this latter hypothesis in their study 
by examining individual performance to see whether or not individuals 
who answered a particular Knowledge question incorrectly tend, in 
general, to miss all higher level questions related to the same informa- 
tion background. They did, however, mention that low scorers on the 
Knowledge subtest tended to be low scorers on all subtests. An alter- 
native hypothesis which might he posed is whether or not those people 
who misinterpreted a particular knowledge question are more likely to 
miss a high level item from the same background if one of the foils 
contained the same misinterpretation than if it did not. Although this 
latter alternative presents an hypothesis which is beyond the scope of 
this present study, it is more in keeping with the possibility of the 
influence of systematic choice behavior on response selection as 
developed here, than is the former hypothesis. 

It is true that for dichotomous variables, the discrimination 
is maximized for items of middle difficulty. In general, if all 
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alternatives arc to "be considered, discriminate on is maximized if the 
selection frequency for all response alternatives on .any item is equal e 
Thus, for a four-alternative item, discrimination is maximized when the 
difficulty is 0 25 when all four categories are used 0 Tn the case of 
forcing a dichotomy on a polychotomous variable, the fact that 50 per 
cent of the examinees get the item right means that the distribution of 
answers on this item is not the product of chance, at least for the 
right answers o The same conclusions may be true for wrong answers, as 
Powell ( 1968 ) has shown, when higher mental processes are involved*. 

For these reasons, it may be reasonable to ignore item difficulty 
except for very easy or very difficult items as a criterion of item 
discrimination - At least the former argument with respect to ascending 
complexity, and the related ethical problem of predetermination of 
hypothetical results were the basis for Kropp et al (1966) ignoring 
item difficulty in the preparation of their tests® The latter argument 
with respect to the discriminative power of polychotomous items, except 
in extreme cases, was the basis for minimizing the importance attributed 
to item difficulty in the present study. 

As Kropp et al (l c j66) point out £p. 3X1 additional problem 

with respect to item analysis arises in the interpretation of correla- 
tions on data derived from taxonomic tests 0 Since the subtests are 
assumed to be hierarchically interdependent , the bivariate distributions 
of scores between subtests appear triangular, making the distribution of 
each higher level skewed further to the right 0 On this basis the 
total-correct score may not be normally distributed for tests of 
practical length used on groups of usual size, hence the use of biserial 
correlation for the validation of an item against the total test score 
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may be inappropriate. This fact, as they point oat, also raises 
problems in the interpretation of any correlation coefficient in their 
study. When determining the discrimination coefficient Kropp e_t al 
(1966) used the traditional procedure. 

Proble ms whi oh a is s t? from impli ci t as sump tions in the Kropp , 
Stoker , and Bash ay; study c The central assumption of the Taxonom y is 
that each higher level subsumes all lov/er levels and adds characteris- 
tics of its own. For this reason, Kropp et al (1966) approached their 
analysis with the implicit assumption that the complexity dimension was 
characteristic of the Taxonomy as a whole rather than being a 
characteristic of each level of subcategory within the Taxonomy . The 
results of their findings with respect to this assumption were incon- 
clusive o Analysis of the subtest scores showed that the order of the 
levels of the Taxonomy as a hierarchy did not fall consistently into 
the order hypothesized. On the other hand, Powell and Isbister (1969) 
tested the assumption that hierarchical categories should be obliquely 
related. Their finding, however, was that the use of a promax rotation 
did not improve the resolution of the factors when right and wrong 
answers were combined, thus the expected obliqueness did not occur 0 

It has already be indicated on page 21 that the Evaluation 
category can occupy most positions above the Knowledge level. The 
Kropp et al (i960) study also found that on the basis of cognitive 
attributes no single category is consistently defined for all grade 
levels tested although the tasks themselves were identical for all 
grade levels. These two findings of Kropp e_t al are inconsistent with 
the Taxonomy as defined. Perhaps the Taxonomy is actually a descrip- 


tion of some of the strategies employed by humans in problem-solving 
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situations. There may be a hiera.rch.lal order to these strategies but 
they may not be taxonomic in IT com 1 s sense of the term. 

A problem which is a Synthesis level problem for a five-year 
old, may be a comprehension level problem for a tv/elve-year old. In 
this context two deviations from this taxonomy would be expected with 
Bloom’s Taxonomy, First, each category of the Taxonomy, with the 
possible exception of ’’Knowledge,” should be characterized by a range of 
complexity levels within the category in addition to an order of 
complexity levels between categories. In such circumstances, as Kropp 
et al (i 960 ) demonstrate, a wide range of possible orders may occur 
among specific samples from category levels. In addition to the most 
common, and expected, order of the categories found in their study, the 
categories occur at least once in any one of three other orders under 
Simplex analysis. 

Second, the strategies involved at different developmental 
stages will vary in accordance with the information and strategy back- 
grounds of the individuals at these stages. For this reason, striking 
dissimilarities in the cognitive attribute content on the basis of the 
Kit of Reference Tests (French, Ikstrorn, and Price, 1963) for any 
category would be expected at different developmental levels. This is 
precisely what Kropp et al fo\md. An important characteristic of this 
change should be its movement toward simplicity. For instance, if we 
reclassify the Kropp et al (1966) ’’Knowledge” category as a ’’search” 
category on the basis of the Undefined factor, w r e find that for grade 
nine the positively correlated cognitive aptitudes are Word Arrangement, 
Letter Sets, Symbol Production which suggests that the grade nines may 
be generating their search strategies as they proceed with the test. 
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For the grade twelves the positive attribute is Locations which suggests 
a more simple and direct approach. 

Another factor which Kropp e_t al (1966) discussed is that the 
difficulty of a problem may be affected by the complexity of the problem. 
It also may be affected by the familiarity or obscurity of the informa- 
tion background and/or strategies required by the problem solver. It 
may also be affected by the nature and the fineness of the discrimina- 
tions which the solution to the problem requires. This latter aspect 
may be related to the nature of the foils. Kropp et al (1966) deal on]y 
briefly with the difficulty problem, /of pp. 90 and 15 . 2 / • 

Contributions of the Kropp , St oker , and B ashaw Study to the 
present study . It may be possible to assume that Bloom’s Taxonomy is 
not a subsumptive taxonomy. In this case, the Kropp ejt al (1966) study 
more strongly supports the possibility of the transcendence of process 
over content than their interpretation of their findings suggests. This 
transcendence of process over content has also been supported by Furth 
(1966) in his work with the congenitally deaf. 

In combination with the other research already discussed 
(see p. 16) there w ould seem to be at least three variables which 
contribute to the choice behavior of examinees on multiple choice 
achievement tests. These are: l) content, 2) process, and 3) effective 

complexity. A fourth possible variable is item difficulty (see; p. 26). 
Since misinterpretation of content and inappropriate selection of 
strategies might both be expected to lead to the selection of an inappro- 
priate response, it is reasonable to assume that at least some students 
will display systematic wrong-answer selection,. Hence, in a forced- 
choice situation the nature of the alternative choice provided would 
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be expected to influence the nature of the selections made. If foils 
are deliberately designed to reflect probable misinterpretations of 
content, or probable inappropriate selections of strategy, more than 
the "right" answers might be used to determine the present achievement 
status of the examinee. 

How can tests which meet these criteria be developed? It is 
fairly clear from the Kropp et al (1966) study that the use of Bloom* s 
Taxonomy is useful as a set of guidelines for the construction of the 
relationship between the stem and the right answer for each item. A 
discussion of common recommendations for the development of the foils 
for each item is presented in the following section. 

Recommendations for Construction of Foils 

The following discussion reviews what some textbook authors 
have had to say to teachers about the construction of foils for multiple 
choice items. Among these authors, Ross and Stanley (195*0 list 
fourteen rules for the construction of multiple choice items. Of these 
only two deal specifically with foil (distractor) construction. 

6. Make all responses plausible 

9. To measure higher levels of understanding, increase the 
homogeneity of the options provided /p. 185 /• 

These authors do not define plausibility , and the example they 
use for increasing the homogeneity of options actually illustrates 
increasing the content specificity of the item. Their second suggestion 
involves increasing the fineness of discrimination between alternatives 

which may be more related to the difficulty of the item than to "higher 

n 


mental processes. 
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As another example, Thorndike and Hagen (I96l) in their second 
edition list ten "maxims for multiple choice items.” Four of these 
have direct bearing on foil construction. Quoting the original we 
find (italics in original): 

4. Be sure that The re is One and Only. One Correct or Clearly 
Bes t Answer . 

5* Beware of Clang Associations . 

So Beware of the Use of One Fair of Opposi tes as Options If 
One of the Pair is the Correct or Best Ans wer . 

9. Beware of the Use of " None of These , " " None of the Above," 
and " All of the Abo ve" as Options . /up . 7^? 75 , 7 6, and p. 77 / 

Whether or not there should be more than one "correct" answer 

will depend upon whether or not the examiner wishes to discriminate 
between levels of insight into a particular problem as in the "best 
answer"- type of test. However, to make such discriminations may require 
the use of information from more than one alternative of any item. 

One of Hoffmann’s (1962) most damning criticisms of the multiple 
choice types of tests arises from the arbitrary assignment of only one 
alternative of the response set to the "right" category in tests of this 
type in such a way as to discriminate against the thoughtful, well 
informed student. This admonition is only appropriate if we are to 
assume that the only answer to be taken into account for any particular 
item is the one designated as "right" whereas the "rightness" may be 
arranged on a continuum in the "best answer" type of test. 

Thorndike and Hagen (l9ol) quite rightly point out that Clang 
Associations (see number 5? P* 28) between stern and right answer tend 
to give the answer away. However, using superficial associations be- 
tween the stem and the wrong answers may in some circumstances be an 
effective discriminating device (see) "Word-Word Link," p c 38). 
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Th orn d i ke an d Hagen f s (19 6 1 ) a. 1 1 e ma. t i v e s , numb e re d e i gh t an d 
nine (see p. 28) are interesting in that they suggest that certain 
aspects of the logi cal re let i onships between answers and foils should 
be considered in foil construction. If a student selects an answer 
belonging to a set described in number five (see p. 28) that is 
logically opposite to the "right answer," then this selection in itself 
may contain useful information. Such a selection reveals at the very 
least which students completely misunderstand the relationship in 
question. Why this sort of alternative should be discarded v/ithout 
qualifications is therefore not clear. The criticism these authors 
make of the "all of these," "none of these" type of alternatives have a 
similar basis. They neglect to say that if "none of these" is correct 
it may be regarded as being logically equivalent to an omission. The 
"none of these" provides a noncommittal response which has the effect 
of making closed-choice alternatives into open-ended alternatives. For 
some purposes it may be useful to knew if the student made one of the 
less common errors, if there are more possible errors than the foils 
account for* In addition, omissions at the end of the paper can also 
mean "not finished." Since there is more than one possible reason for 
omitting an item, interpretations of an omitted response becomes ambig- 
uous. For these reasons, the basis upon which a student makes a non- 
committal response may be a valid question for study. 

More recently, Ebe'l (1965) lists 48 "suggestions for preparing 
good multiple choice test items." Of these 48 only five directly 
relate to foils or "distractors . " He also rates these as "desirable" 
or "undesirable." Quoting the original: 

32. Item using true statements as dis tr a ctors o (Desirable) 
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0 0 

Item 

using stereotypes in distra. 

* t or s 0 ( De s i rab le ) 

34 . 

Item 

using ob 3 cu re d i s t r a 0 tors. 

(Undesirable ) 

33 . 

Item 

using a highly implausible < 

’i i 3 1 ra c tor 0 ( Und e s i rab le ) 

36c 

Item 

involving verbal trick, /'p; 

). 183-18 3 / (Undesirable) 


The first tv/o of these are examples of the use of errors in 
logic which Sanders fl 966, p. 104^7 suggests we teach the students to 
recognize, hut does not elaborate on, with respect to measurement. 

Ebel's ( 1965 ) suggestion numbered 34 immediately above, proposes 
that the use of obscure or complex vocabulary is undesirable. On the 
contrary, if the intention of the examiner is to study responses to 
obscure, ambiguous or complex situations, this type of item may be 
desirable o Although other methods may have certain advantages when 
measuring complex human behavior, the multiple choice method retains two 
particular advantages. First, a high level of control can be maintained 
in the alternatives supplied so that the ’’controlled sample of perform- 
ance” characteristic of all tests can be very explicit. Second, once 
the performance components of the complex behavior which is to be 
observed has been established, accurate counts of the frequency of the 
choices which fit the categories of alternative (whether right or w rong) 
designed to measure these components is a simple matter. Other meas- 
uring instruments have other advantages at the expense of these two. 

The study of such items would probably necessitate examining all 
responses to each item. Thus, Ebel's ( 1965 ) proposal that this type of 
item is undesirable can be considered valid only if the ’’one right 
answer” assumption is considered valid. Closer scrutiny of this entire 
problem seems reasonable. 

In suggestion numbered 35? (p. 30 ) . relating to implausibility 
the problem of a definition for plausibility arises once agairio 
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Plausibility may be a function of the rationale used in determining the 
construct and content validity of the test. The examiner must be able 
to anticipate what alternatives may be plausible to the examinees. 
Without a definition of plausibility, implausibility is impossible to 
determine. In fact, plausibility is often defined on a post hoc basis 
from the item analysis with foils having a low selection ratio being 
classified as "implausible. n However, if the purpose of discrimination 
is to identify individuals for differential treatment a foil which 
identifies ten or twelve out of 1,000 students may be more valuable than 
one which identifies 250 students. 

Finally, many foils which seem to involve a "verbal trick" may 
have a valid function. These verbal tricks are probably of three kinds. 
The first kind could be the introduction of a peculiarity of wording 
designed to produce interpretive or misreading errors on the part, of 
some students. The second kind of "verbal trick" is found in such 
things as Zeno’s Paradoxes (c; 340-264 B. C.) in which the "verbal 
tricks" involve a faulty assumption in the reasoningo The third kind of 
"verbal trick" introduces the possibility of detecting in the examinee 
an inappropriate "set" for the correct solution of the problem. Both 
the Einstellung effect and "functional fixity" may possibly be used to 
develop examples of foils for this type. In each of these cases it is 
conceivable that the information generated from response to these types 
of item could have discriminative value. The issue here, once again, 
is both contend and construct validity. Bee s the "verbal trick" give 
the intended information, or interfere with the obtaining of this 
inf ormationo 

V/e find this same ambiguity of advice prevailing throughout the 
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range of standard texts in this area*, From the ETS booklet Mu I t iple 
Cho ice Questions ; A clos e look (1963) through to such writers as 
Ahmann and Clock ( 1963)5 Gronlund (196'5) and Kell ( 1965 ) we find the 
great bulk of the suggestions about item writing discussing the func- 
tional, linguistic, and structural characteristic's of the stem, and 
stem-right answer relationships, with only minimal and often contra- 
dictory treatment of the foils and how to construct them. 


Need for a Basis for Interpreting Foi 1 Selectio n 

On the basis of the above discussion we can identify several 
general bases for foil construction as presented to constructors of 
multiple choice tests. These are: 

1. Logical Relationships 

2. Logical Errors 

3. Partial Information 

4 . Mi s inf orrna t i on 

5# Obscure Relationships 

6 . Misunderstand ing 

7o Verbal Tricks 

Not all of these are regarded favourably by the authors 
mentioned nor are these bases consistent with themselves or between one 
author and another. It would seem that the recommendations have been 
developed on a trial-and-error basis derived from the experience of 
professional test constructors during their attempts to meet the 
statistical criteria of an "effective item." 


The Possible Value of the Experl mental Test 

The testing technique being explored in this study hypothesizes 


33 


that Bloom* s Taxonomy adequately describes the strategies leading to 
right answers, and that a set of logically based guidelines for foil 
development effectively describes some of the possible systematic 
deviations from the ideal outcomes of these strategies o These two 
facets combine to form the construct characteristics of the testing 
techniques under study 0 Of course, any findings from a purely explor- 
atory study must be tentative 0 However, wrong answers from a "strategy 
test may increase the predictive power of that test for total achieve- 
ment scores (found in the usual way) from independent achievement tests 
In this case, more than the information background of a test may be 
involved in "success" on an achievement test c . Such findings would 
strengthen the support for the hypothesis that process may transcend 
content. Furthermore, this study may suggest some of the typical types 
of errors students may make as they mature intellectually which might 
eventually lead to the establishment of a behavioral description of 
development which is independent of test content, and of educational 
strategies which may be appropriate to the stages and phases of this 
developmental sequence 0 

On the application side, the main advantages of guidelines for 
foil construction would be expected to involve the l) simplification 
of item writing, 2) clarification of why a foil is wrong, and 3) 
possibility cf producing diagnostic tests in subjective content areas,, 


CHAPTER III 


RESIGN OF THE EXPERIMENTAL TEST 


From the discussion developed in Chapter II, the usefulness of 
Bloom’s Taxonomy for the development of process-oriented items was 
suggested. The possibility that Bloom’s Taxonomy may not display the 
assumed sub sump tive characteristic between categories does not minimize 
its role relative to the establishment of the construct validity of a 
test. The evidence presented suggested that there is no similar set of 
internally- consistent construct guidelines for the development of foils. 
Seven general categories of foil based upon recommendations from the 
literature could be established. Using these seven as a starting point 
the first task in this chapter is to develop a systematic set of Guide- 
lines which may prove helpful for foil construction*. The seven cat- 
egories can be further reduced. Perhaps the most important category 
involving strategies are those foils which can be based on probable 
errors in logic ma.de by the examinee. Partial information can lead to 
an error in logic if the wrong strategy is used to generate the missing 
information o It can lead, also, along with other casues, to an over- 
simplification of the problem. Since only the product of the choice- 
behavior is observable on a multiple choice test, it would be reasonable 
to include Logical Errors, Partial Information, some Verbal Tricks, and 
perhaps some Logical Relationships (like answers which are opposite to 
the right ones) in a list of categories of foil generation where higher 
mental processes are to be tested. 

Misinformation and misunderstanding may be identical or they may 
be different in that the misunderstanding may be related to the reading 
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of a specific item or group of items rather than to a weakness in the 
information background of the examinee. If the examinee succumbs to 
certain kinds of verbal tricks (for example, the use of meaningless 
jargon in a foil) his problem may be more immediately test-related than 
background-related provided that he is not misled elsewhere on the test 
when jargon is not used. In the present context there is the possi- 
bility that in addition to the process variables there may be a class 
of foil related to the linguistic characteristics of the item. This 
class of foil may be designated as a "Misreading” class = 

Another possibility is that the examinee has systematically 
misclassif ied a particular piece or set of information. In items based 
on the possible logical relationships among the total information back- 
ground this piece of misinformation will lead to the systematic 
selection of specific wrong answers each time this misclassif ication 
appears in a foil. For instance, the person who confuses the work of 
Kebb with the work of Hull. Foils of this type, and of several others, 
are beyond the scope of the present study and are, therefore, classified 
in the "Others" class. Subsequent research may be expected to elaborate 
this latter problem. 

The Guidelines for Foil Construction 

Our earlier discussion showed that similarities can be found 
between common errors on nonverbal tests and verbal tests (see: p. 17). 

The present experimental test in its original version was based on 
classifications involving logical fallacies and logical relations. The 
test has been revised in an attempt to improve item discrimination for 
the present study. The same reading selections and general questions 
and overall format remained unchanged. The Guidelines which follow 1 
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were used to revise the foils. 

Since the definitions of foil classes, as they were originally 
used, tended to lack precision, they were redeveloped for this study • 
The Guidelines are described below ( see p. 37) in terms of the proce- 
dure used for constructing each type of foil. Four classes were 
produced : 

1. Strategy class ; the largest group of Guidelines to be 
developed for this study is based on the logical 
characteristics of the foil relative to the right answer 
and information background. Because these types of foil 
are suggestive of incorrect analytic procedures they are 
collectively referred to as the "strategy class." 

2c Misreading class ; this group of foils is based on semantic 
characteristics of the foils relative to the right answer 
and information background. The nature of this test, 
i.e. an open book test, would be expected to reduce the 
possible number of foil categories in the misreading class 
because an examinee who feels he has misread an item can 
refer directly back to the information background supplied. 
This class of foil probably has many more members which 
would describe different aspects of misreading where 
information recall is the source of information. An 
example of this situation is the Jargon (j) category 
(see p . 39) . 

3. Other ; the foils in this class are unclassif iabie , at least 
by the present Guidelines. Future studies are expected to 


reduce but not eliminate this class of foils. 


4o Mis Inf oriria t ;i. o n class ; the nature of the experimental 

examination, (ice. an open book examination) precludes the 
development of misinformation foils which would be expected 
to occur in the context of a test requiring information 
recall o These would be expected to be related to "Knowledge" 
level items, a level of item which was not used in this 
examination for reasons already discussed (see p. 19ff).. 

The first two of these major classes may be subdivided on the 
basis of a specific description of how a foil which fits any particular 
category is produced. This subdivision follows: 

Guidelines 
A. Strategy Class 

1. Ov e rgerie ra 1 i x a t i on . (OG) In the development of this type of 
foil the author retains the correct relationship of the 
best answer in its entirety and adds some irrelevant 
information. (For example, see item 1A, p. 15*0 * 

2. Oversimplification . (OS) In this case the author omits one 
or more parts of the best answer. (For example, see 

item 20, p e 156) o 

3. Inversion (inv). In this case the author makes a statement 
in some way opposite to the best answer c (For example, 

see item 4C, p. 158). 

4o Irrelevancy (irr). In this case the author makes a true 
statement which is unrelated to the best answer, or a 
statement which could be a correct answer. (Perhaps by 
virtue of some restriction in the stem) 0 (For example, 
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5. Invalid Assumption (lA). In this case the author begins 
with an unwarranted assumption about the background or 
solution to the problem and thus writes a foil which would 
be correct as if this assumption were valid. (For example, 
see item 1C, p. 15^-) « 

6 . Subs t i tuti o n (Sub). In this case the author replaces at 
least one of the elements or the relationships of the best 
answer by a corresponding element which is less acceptable. 
(For example, see item 2B, p. 156)* 

7. Transposition (Tr). In this case the author modifies the 
order of the elements in an ordinally dependent relationship. 
(For example, see item 30 C, p. I 8 l ) 0 

8 . Commoii Misconception (CM). In this case the author utilizes 
his knowledge of the probable common misconceptions held by 
the examinees to write the foil. (For example, see item 5B, 
p. 'l59)o 

B . Mi sreadlng Class 

1. Word- Word Link (WW). In this case the author produces a 
false statement which has strong verbal links with the stem 
or background information by either repetition or associa- 
tion. This type of foil may be similar to Foulde’s (1965) 
Class I) error, see p. l 6 0 (For example, see item 7B, p 0 162)0 

2. Redefining of Te rms (RT). In this case the author uses a 

Li 

This type of foil misleads certain of the best students, 
perhaps the more imaginative ones (see p. 13). 
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word or words in the foil in different literal or 
connotative sense than it is used in the stem or background 
information. (Per example,, see item 11 D, p Q 1 66) 0 
3 ° Jargon (j)„ In this case the author produces a quasi - 

meaningful statement which tenuously relates in some manner 
to the best answer. The use of coined "near words" may also 
be present. (Not used in experimental test; see p Q 3o)* 

C. Others 

1. Other s (0 ) 0 In this case the foil is, at present, for some 
reason , unclassi liable e 

These are the Guidelines which were used in the construction of 
the foils in the experimental test* 

Structure of the Experimental Test 

As already mentioned, Bloom 1 s Taxonomy was used as a guide for 
the construction of the stem and right answers of the experimental test. 
An interrater reliability between judges for the advance classification 
of right answers was reasonably high (r - 083 ). The Guidelines just 
given on pages 37-39 were used to construct the foils. The interrater 
reliability for foil classification was somewhat lower (r = 0 6 2 , N - 5)* 
As in the case of Bloom* s Taxonomy for the right answers, the 
Guidelines presented the immediately evident advantage of increasing the 
number of possible foils which could be considered for any one item, 
making foils easier to generate than they were in the more usual "hit- 
and-miss" method. An additional advantage for the Guidelines became 
evident after the earlier atimini strati on of the test. The Guidelines 


40 

help clarify the "basis for why any foil should he considered vrongo The 
absence of such a basis is a common weakness of teacher-made tests 0 

The examination consisted of five short reading selections drawn 
from material which was in some way related to educational psychology 
since ‘this was the central topic of the course in which this examination 
was to be used. They were also chosen on the basis that it was rel- 
atively unlikely for the examinees to have encountered the works from 
which these selections were drawn in their previous training. To the 
extent that these selections were specifically oriented to the vocab- 
ulary of the studies of psychology and education, this test demanded 
information recall from the examinees. Aside from this restriction, it 
was assumed that all items could be answered correctly solely upon the 
basis of the information given in these selections. This assumption 
may not have been entirely warranted 0 

Since most of the necessary background information was assumed 
to have been supplied in the test, no Knowledge category items were 
generated. On this basis, the test was intended to be a 1 'higher mental 
processes ,f test c Since the major emphasis of the test involved logical 
analysis, it was assumed that the test was essentially an "Analysis" 
level testo The findings of the preliminary version as reported in 
Powell and Isbister (1969) confirm this assumption® 

Content ana Construct Oh arac t eri sties of the Experimental Test 

A detailed item-by-item discussion of the test may be found in 
Appendix B (see: p 0 153 ff)o In brief, five reading selections related 

to the area of educational psychology were chosen on the basis of 
information density and the unlikelihood of the examinees having 
encountered the selections previously® These selections which are 
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both given and referenced in Appendix B are referred to subsequently ass 
lo Stupidity 
2o Awareness 
3. Aggression 
4 0 Discipline 
5o Progress 

The 30 items in the test were classified using Bloom* s Taxonomy 
as a construct model as indicated in Table 1 (p. 42) and elaborated in 
the discussions in Appendix I3o Wo Knowledge -level items were aeveloped 0 
Items were classified as Synthesis if they required the examinee to 
organize the material from more than one reading selection into some 
systematic relationship when deciding which alternative to select for an 
answer. A reasonably high interrater reliability (r = 083 ) was found for 
the classification of the items based upon the item format, the stem, and 
the stem-right relationship when the right answer was indicated 0 Dis- 
agreement occurred among several raters, and among other reviewers of 
this study on the keying of some of the items 0 This disagreement would 
be expected from the subjective nature of the content and the related 
differences among the value systems inherent in any group 0 

Much less agreement among raters was found for the foil 

CL 

classification (r = „62)". This poor result was expected for the 


word of caution is in order 0 This low interrater reliability 
suggests that readers following the item-by-item discussion in 
Appendix B may disagree with the foil classification given and 
with the reasoning behind it. It would, be interesting for the 
reader to record his disagreements and to compare these with the 
results of the cluster analysis as given in Table II, page 73? 
and Table 12, page 77. 
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TABLE 1 

CLASSIFICATION OF ITEMS 
USING BLOOM'S T AXONOMY 
(30 ITEMS) 



Comprehens i on 

Bloom 1 s 
Appl i cat ion 

Category 

Analysis 

Synthesis 

Evaluation 

Item 

Numbers 

3,6,12,15 

5,7,11 

1,2, 4, 8, 9 

13,14 

10,16,18 




17,19,20,21 

22,23,24,25,27 

26,28 

29,30 


Totals 

4 

3 

14 

6 

3 


reasons already given (see: pp c 4-5)? and the findings of Kropp et al 

( 1966 ) which suggest multiple interpretati ons of specific items and 
alternatives within heterogeneous groups 0 This multiplicity would be 
expected to increase with the complexity and subjectivity of the content 
so that a high level of agreement, even among professionals on the 
particular test used in this study, would be unlikely. 

To illustrate the extent of this problem, a check was conductedo 
One of the raters of the items disagreed with the classification of 
three foils in particular. Of these three only one of his reclassifi- 
cations was supported by the cluster analysis as given in the results of 
the study (see: p Q 182 for details) 0 This one-in-three success ratio 

was equivalent to that of the experimenter 0 

The overall appearance of the experimental test suggests that in 
the traditional sense it is a very poor one. The internal consistency 
value for the test was K-R 20 = .34. A review^ of the item difficulties 
and biserials from Table 40 of Appendix A (p. ipO) is equally discour- 
aging. However, the use of Bloom as a model for the right answers and 
the Guidelines for the wrong answers suggests that the test should not 
be considered homogeneous. For this reason, and the reasons given 
earlier when discussing this same problem relevant to the Kropp et al 
( 1966 ) study, (see p u 21 ff) the use of traditional evaluative proce- 
dures on this test may be questionable 0 Support for this position is 
found in the Procrustes rotation of the factors to fit the clusters 
which gives six nearly orthogonal factors which display quite adequate 
internal consistency (see: p. 70 ). 

The foil classification procedure differed from the item 
procedure in two important respects. First, although the Guidelines 


were also used as a model for the possible in forma ti on content of the 
foils, the relationship between this model raid any possible characteris- 
tics of the examinees, in the almost total absence of research, was 
largely unknown 0 Second, whether these Guidelines formed mutually ex- 
clusive categories, or a hierarchy paralleling Bloom could only he 
inferred from the assumptions which went into their development 0 

These foil Guidelines were used to help develop the foils as 
well as to classify them 0 A summary of the classification of the foils 
is given in Table 2 „ 


TABLE 2 

CLASSIFICATION OF WRONG ANSWERS 
USING THE FOIL GUIDELINES 


Classification of Foil 

A B C 

Strategy Misreading Other 


OG 

OS 

Inv 

Trr 

IA 

Sub 

Tr CM 

ww 

RT 

0 

1A 

2C 

4C 

ID 

1C 

2B 

30C 5B 

7B 

11D 

All 

4A 

3D 

8D 

2D 

5A 

3A 

9D 

10D 

15D 

foils 

6 B 

5C 

9B 

3C 

8A 

6c 

10A 

13D 

29D 

in 

8C 

7D 

10C 

4D 

13A 

7 A 

12B 

15C 

30D 

items 

12C 

11C 

11A 

6d 

25D 

90 

13C 

30A 


19-24 

14D 

26c 

12D 

14A 


14C 

16D 



26B 

15A 

29C 

17C 

16A 

27A 

17D 

27D 




16 c 



18C 

28D 

18A 

29 A 




17B 



230 


25 A 






18D 26D 

280 270 

28B 


11* * 7 ? 12 7 9 1 8 5 4 19 


* = Frequency of foil in each category 
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The test used in this study is a revision of the one reported 
in Powell and Isbister (l 969) which had a slightly different purpose c 
The present discussion supported by the item-by-item analysis given in 
Appendix B would seem to demonstrate that, Cor all the faults of the 
instrument, the content and construct requirements for this test as 
laid out in Chapter II have been met to a reasonable degree of 
acceptability. 

In the Powell and Isbister (1969) study the advance 
classification was taken as given and profile scores we re developed 
accordingly® The resulting score sets were treated as independent 
variables and subjected to principal axis factor analysis in order to 
determine relationships among these scores 0 In this study the advance 
classification was not taken as given but subjected to a comparison 
with a cluster analysis based upon the relationships found among each 
of the alternatives o In this present study the acceptability of the 
advance classification system as exemplified in the test was being 
studied. 

On the basis of what has already been said about the problems 
that communications of this type produce, it would be reasonable to 
expect the advance classification systems used in this study would 
not hold up without the qualifications derived from the possibility of 
l) multiple interpretations of the communicating stimulus, 2) multiple 
methods of integrating mid relating the stimulus to each individual’s 
own experience, and 3) leading to multiple interpretations of the 


responses 


CHAPTER TV 

THE DESIGN OF THE STUDY 

The success of this study is contingent upon three aspects 0 
First, the study must stand upon the acceptance of the logic of the 
content and construct validity of the experimental test as given in 
Chapter II I 0 

Second, the construct validity must find evidential support in 
the statistical results of the analysis of the examinee performance on 
an administration of the experimental test 0 This support can be found 
in several ways G First, the advance classification may be found to re- 
appear in the statistical patterns. Second, the content pattern might 
be shown not to be an important contributor to the statistical patterns 0 
Third, in the event that the advance classification cannot be supported, 
some reasonable method of modifying the advance classification which does 
not violate the construct assumptions, such as possible multiple 
interpretation of the items, may be found 6 Fourth, the patterns should 
cross-validate between equivalent independent groups. Fifth, if cross- 
validation fails, a reasonable explanation which fits the data and the 
construct assumptions must be found to explain this failure 0 

Third, however much the construct validity is supported, v/rcng 
answers in some form must also contribute significantly to the prediction 
of achievement scores obtained in the usual manner before they can be 
considered to contain achievement information 0 

These three aspects form, in combination, the necessary and 
sufficient conditions needed to demonstrate that the method of test 


construction used in this study can be used to develop tests which 
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contain useful information about student performarr: e in the answers 
given to the foils o A further restriction tc this problem arose. Since 
the study began with categorical data, it should end with categorical 
interpretations in so far as is possible. 

To begin with, however, the answer selections on the 
experimental test cannot be assumed to have any of the usual continuous 
distributions. The selection pattern can be considered to be cat- 
ego ri cal 5 since one choice is made for each item, but not dichotomous. 

An expedient method of defining categorical data mathematically 
is to treat categorical membership as "one" (l) and nonmembership as 
"zero"' (0). A matrix of categorical data should have the following 
properties : 

1. The centroids of normalized clusters from the matrix should 
tend tc he either orthogonal or opposite each other. 

2. The orthogonal projections of the members of a cluster upon 
its centroid should be near unity,, 

3. The orthogonal projections of the members of a cluster upon 
the centroids of all other clusters should either be non- 
existent, or nonsignificant. 

Figure 1 illustrates a typical response matrix which displays 
these properties for the twelve variables included, and may display 
these properties for some reduction of the matrix to less than twelve 
variableso Figure 1, a sample response matrix is on page 48. 

The usual procedure for test analysis is to use the right 
answer portion of Part B and Part C (the total number correct) and 
to treat the wrong answer division of Part B as redundant. 

If- all four alternatives of each item are considered the 


Student 

Number 


part A 


Item Alternative 


Right 
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- Correct response 


FIGURE 1 

A SAMPLE RESPONSE MATRIX 


statistical problem of linear dependency arises. To illustrate what 
is meant by linear dependency refer to Figure 1 above c Notice that in 
Part A of this figure the sum of each row is always three. Only in the 
case of omitted items will the sum of the answers be less than the 
number of items. 

Since all alternatives are being counted, this total is 
predetermined as being the number of items. If the columns are added 
vertically, the sum of the columns within an item is predetermined at 
the number of examinees® In the case of Figure 1 above, this sum is 5- 
In many statistical procedures, linear dependencies have the effect of 
rendering indeterminate or ncn-unique solutions® 


The solution to this problem used in this study was to partition 
the matrix as indicated in part B ol Figure 1 (see: p 5 48). Two 
matrices, one for the right answers and one for the wrong answers, were 
made from the original response matrix. The categorical property was 
retained within each of these two new matrices. This procedure had the 
effect of treating right and wrong answers as though they were 
independent. 

Part G of Figure l*(see p. 48) shows the row sum (horizontally) 
of the right answer partition of Part B. This sum, which is the total- 
correct score, is the usual approach to the interpretation of test 
results. It is with Part C that the results of the statistical analyses 
of Part B are being compared. 

There are several possible methods of dealing with categorical 
data*, Since this study is concerned with relations among categories 
the most reasonable approach is to begin with ph i correlation co- 
efficients between the category pairs. This procedure produced two 
correlation matrices, one 30 by 30 for the right answers, and one 90 
by 90 for the wrong answers. 

Since the results of these analyses were to be cross-validated, 
the original group of examinees were subdivided by rand on assignment 
into two groups (Group A and Group B) 0 The data for both groups were 
subjected to the same statistical treatment although most of the 
interpretive work was done with the results from Group A. 

The result of this latter subdivision was that the analytical 
aspects of this study began with four phi coefficient matrices (one for 
right answers and one for wrong answers for each of Groups A and B). 
These four matrices were the basic data for much of this study*. They 
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may be found in Tables 32 to 39 of Appendix A. 

The phi matrices gave relationships among pairs of alternatives 
cnly 0 To proceed further, it became necessary to find relationships 
among these relat j onships 0 From the original structure of the experi- 
mental test there were two patterns of relationship which could be 
sought. The first was the pattern as defined by the advance classifi- 
cation based on Bloom’s Tax onom y, the second was the pattern as defined 
by the content (information background) of the items and foils 0 

One of the methods of checking the data for these patterns 
which could be used is the Procrustes rotation solution to factor 
analysiSo The procedure began with the principal axis factor solution 
and found the best rotation of this solution in a least squares sense 
for a given matrix. 

A factor solution was used to remove as much measurement error 
as possible from the further analytic procedures used in this study 0 
The phi coefficient is extremely sensitive to the marginal proportions, 
particularly when the selection ratios deviate considerably from . as 
in this study where four alternatives are being used. Slight changes 
can have a profound effect upon specific coefficients 0 This effect can 
be reduced by the factoring procedure which takes tne relations among 
coefficients into account 0 

The principal axis solution was used to get as much variance as 
possible in as few factors as was reasonable 0 

A third approach normalized the principal axis matrix by rows 
and then found the distances between the ends of the resultant vectors 
by the usual distance formula. Clusters were then defined in terms of 
minimizing within cluster distances and maximizing between cluster 
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distances o The mathematical procedure used in this study is given in 
Appendix A (see: Table 4l, p c 153 )o 

The advantage of this procedure is that if a good fit is obtained 
the solution alleviates many of the problems of rotation which are other- 
wise inherent in factor analytic solutions 0 

All three of these solutions can lead to results which can be 
interpreted categorically o If a good fit is found with the target matrix 
for advance classification either by process or by content, then the cat- 
egories of the original classification were to be used® If, on the other 
hand, good fits were not found, then the categorical solution of the 
cluster analysis procedure would be studied for possible interpretation 
on the basis of either process or content 0 In this latter case, the data 
would also have to show that there were no contradictions to the content 
or construct assumptions as given in Chapter II, (p„ 12 ff) otherwise 
this study would not meet the necessary and sufficient conditions 
required as outlined at the beginning of the present chapter on page 46 c 

An additional advantage of these three procedures is that they 
all begin from a principal axis factor solution of a correlation matrix® 
Pu.rthermore , if the same factor solution is used in all three cases, the 
goodness of fit of the cluster solution can also be found by the 
Procrustes method 0 Hence all three of these solutions can be subjected 
to the same criteria® 

Since the object of this study was to support the construct 
characteristics of the experimental test, the analysis began with an 
attempt to find the best possible cluster solution to this construct 


criteria for the data of Group Ao It was decided that the best possible 
solution would involve having clusters defined by the most frequently 
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recurring category as defined by the advance classification. The number 
of these identifying elements was to be as large as possible for each 
solution, i.e. the right answers and the wrong answers. The number of 
factors in the principal axis solution needed for this result was then 
taken as standard for all solutions involving the same kind of data c 
For instance, six factors gave the best solution for the right answers 
for Group A. Hence, six factors were used for all right answer analyseso 

For cross-validation the identical statistical procedure used 
with Group A was repeated with Group B. Cross-validation was then 
established once again from a best-fit match (in terms of most fre- 
quently recurring members) bet ween the categorical results for Group A 
and Group Bo Several procedures we re used until a satisfactory match 
was found. Once again, the cross-validation could not violate the 
construct considerations outlined in Chapter II, page 12 ff for this 
study to be successful. 

Finally, the categories which were established as being 
potentially meaningful in the earlier parts of the study were used as a 
basis for rescoring the experimental test. The results of these sub- 
test scores were combined in several ways and their ability to predict 
the total-correct scores of two independent achievement tests for the 
same subjects was compared with the predictive power of the total- 
correct score. In this latter case it would be necessary to show that 
the use of wrong answers consistently improved prediction over 
total-correct score and combinations of right answers. 

If all these criteria were met., the value of wrong-answers 
part of performance information would be demonstrated. With so many 
criteria to meet, the probability that such conclusive evidence would 


be found is exceedingly low* On the other hand, trends in the 
directions indicated could be treated as suggestive © The borderlines 
between undemonstrated said suggestive, and suggestive and conclusive, 
are unclear and subject to disagreement© 

Statement of the Problem 

Since this study is exploratory, attempting to demonstrate the 
presence of information in wrong answers and to discover the major 
properties of this information, an elaborate theoretical structure for 
formulating testable hypotheses was considered to be unnecessary© 
Instead, the procedures suggested for the establishment of grounded 
(da ta-based) theory as outlined by Glaser and Strauss (1967) was used© 

Such theory as is used in this study comes from well established 
principles in psychology, communication theory, and test construction 
theory© Beginning with the S-O-R paradigm common 1 y used in problem- 
solving studies, it became evident from communication theory that each 
of the members of this paradigm may best be considered a composite© 

That is, any specific stimulus may be subject to a range of interpreta- 
tions© If this stimulus requires the solution of a problem, the 
specific interpretations may be subject to a range of solution strat- 
egies some of which may lead to "correct 11 and some to "incorrect" 
solutions* In the multiple choice test, the examinee can be expected 
to try to match the alternatives given him in the item within the 
interpretation range and strategy range available to him© In this case, 
the most reasonable assumption would be that most, if not all, responses 
given by an examinee to a multiple choice achievement test would be 
selected on a systematic basis© 

If some characteristic of particular alternatives in two 


54 

separate items are sufficiently similar to the apparent right solution 
.in the view of the examinee, he can he expected to choose both of them 0 
If a sufficiently large number of examinees select this same pair of 
alternatives this joint selection will appear as a high correlation in 
the phi coefficients relating the two events 9 thus becoming '’systematic 11 
in that it would produce a significant statistical event 0 

In the usual procedure used for scoring multiple choice 
achievement tests, only the right answers are treated as systematic ' in 
this sense, hence the requirements usually set for their performance 0 
This study addressed itself to the exploration of the possibility that 
’’most if not all of the answers given to multiple choice achievement 
tests are selected upon a systematic basis," This psychological 
hypothesis is the basic theoretical proposition proposed by this study. 

Since it is possible that wrong answers may influence the way 
in which items behave, and, inferring from communication theory, the 
suggestion emerges that each alternative may have more than one 
interpretation among a group of examinees. Thus, there are four 
possibilities, l) that the systematic characteristics depend upon 
content; 2) that the systematic characteristics depend upon the 
advance classification as defined by the two process models of Bloom 1 s 
Taxonomy and the Guidelines; and 3 ) and 4) that the systematic 
characteristics depend upon multiple interpretations as based upon 
content or process. The study could have no commitment toward any of 
these four possibilities. 

For an exploratory study, Q.E 0 D. can be written at this point 
without further interpretation attempts 0 The developmental characteris- 
tics of wrong answers, their relationships with personality variables, 


etc 


wi th ri gh t answe rs , 
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o exceeds the scope of this dissertation © These 


topics are, of course, legitimate areas for future research 0 


The Sampl e Used 

The experimental test was administered to 277 summer school 
students in a one semester course in educational psychology at the 
senior level. The age group range of these students was from ±9 to 55 
with the median age about 30 ? and most of the students having had some 
teaching experience 0 The overall group was subdivided by random 
assignment into two groups (Group A of 139 students; and Group B of 138 
students) o A t-test for independent samples based on the total-correct 
scores of the experimental test designed to confirm the equivalence of 
the scores of these two groups is reported on page 92 © 


CHAPTER V 


RESULTS and their interpretation 

A somewhat different procedure to the one usually employed was 
adopted for this study. To begin with, the usual procedure for scoring 
and interpreting multiple choice achievement tests is to count the 
number of items each examinee has correct© This procedure is sometimes 
modified by the specification, by various methods, of subtests of the 
total test. One of two general procedures is usually employed. Either 
the experimenter establishes the categories into which the items fall in 
advance of the test administration and then interprets his results on 
this basis, or he groups his results cn the basis of some analytical 
procedure and then endeavours to interpret these groupings 0 Powell and 
Isbister (1969) used the former procedure, and Powell (1968) used the 
latter 0 In general , only right-answer information is used. 

The present study endeavours to link advance classification and 
statistical classification, and also endeavours to use wrong answers as 
well as right answers in the interpretation of test results. As has 
already been indicated, very little research of the type just described 
is present in the literature© For this reason, this study can best be 
described as exploratory in which negative results are more likely to be 
indicated than are positive results© 

Each item on the experimental test had four alternatives, hence 
the study began with four variables for each item 0 The response matrix, 
therefore, contained a "one" (l) for each alternative selected by each 
examinee; otherwise "zero" (o)„ Since the examinee was allowed no more 
than one choice per item for 30 items, each examinee would have a 
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maximum of 30 "ones" in the vector of 120 variables which represented 
his selections o Because these selections were further restricted to one 
in each group of four, each variable was linearly dependent upon the 
other three in the same item. In order to remove these linear depend- 
encies, the performance matrix was partitioned into a right-answer 
matrix and a wrong-answer matrix. These latter two matrices were 
subsequently treated as being independent . 

In order to attempt to cross-validate the findings, the 
examinees we re randomly assigned to two groups, Group A and Group B. 

All the statistical analysis done which was not related to cross- 
validation was performed on the data from Group A. The relationship 
between the mean total-correct scores of Group A and Group B is given 
in Table 

In addition, since a relationship between advance classification 
and statistical ordering was being attempted, an advance classification 
system was used separately for the items as represented by their correct 
alternatives and their foils 0 These classification systems were 
discussed in detail in Chapters II and III 0 

Since an attempt to find a consistent interpretation of 
performance is being made, the examinees were randomly assigned to two 
groups so that the interpretations could be examined for cross- 
validation. Hence, the basic data for this study consists of two phi 
correlation matrices (see: Appendix A, Tables 32 to 39) for each of 

the two groups. The correlation matrices represent the intercorrelation 
between variables across examinees for the right answers and for the 
wrong answers in each group 0 


Finally, two achievement test scores were obtained for each 


examinee. One of them was concurrent .in the sense that the 
experimental test formed a sub test in the mid term examination given in 
a one-semester course. The other achievement score was part of the 
final examination in the same course. This data were collected so that 
the predictive validity of the various interpreted categories and their 
predictive cross-validation could be determined. 

Several steps were taken in each phase of the analysis. For 
instance, attempts were made to interpret the right answers on the basis 
of both factor analysis and inter-point distance cluster analysis. This 
step was followed by a detailed logico-semantic analysis of the right 
answer clusters in an attempt to interpret these clusters 0 

A similar logico-semantic analysis v/as made of the wrong-answer 
clusters. 

Attempts were then made to cross-validate the 'advance 
classification, the interpreted clusters and a particular grouping of 
the interpreted clusters. 

Finally, the predictive validity of the advance classification, 
the interpreted clusters, and the grouped clusters was found 0 This 
validity was found in each case by using the right answers alone, and 
the combination of both right and wrong answers 0 

The discussions which follow adhere to this sequence. 

Interpretation of Right Answers Using Factor Analysis 

On the basis of the advance classification there were two 
possible interpretations based upon either of two independent classifi- 
cation systems with respect to the right answers given by the examinees. 


One of these interpretations could have been best described as a 
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"process” interpretation based upon classification of the items on the 
basis of Bloom’s Taxonomy Tne other possible interpretation was 
"content" in which the items were classified on the basis of the 
information background required to answer them. 

An attempt was made to verify the possible existence of either 
or both of these two interpretations. The primary data for this attempt 
was a six-factor unrotated principal axis factor matrix derived from the 
phi correlations for the right answers. This matrix was rotated by a 
Procrustes solution to find the best fit (in a least squares sense) to 
two target matrices. The first of these targets specified a simple 
structure which indicated the way in which the items were classified 
using Bloom’s Taxonomy ., The second target matrix specified a structure 
which indicated which items referred to each cf the several reading 
selections. The matrix structure was not always simple since some of 
the items referred to more than one selection. 

Table 3 (see: p. 60) gives the target matrix and the pattern on 
the primary axes as related to the "process" classification of these 
items. 


It is evident from the results that the pattern does not 
reproduce the target matrix in any satisfactory manner. This finding 
suggests the conclusion that the advance classification of items using 
Bloom"s Taxonomy did not give a satisfactory indication of the way in 
which each item performed. Table 4 gives the correlation between the 


primary axes in this solution. Table 4 is on page 6l 
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TABLE 3 

PROCRUSTES ROTATION OF THE ADVANCE 
CLASSIFICATION OF RIGHT ANSWERS 
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i 


1.00 


1 11.42 

4.97 



3.16 

20 

i 

i 


1.00 


f f 
I I 

2.40 



2.75 

21 

i 

i 


1.00 


t I 
\ t 

1.02 

1.11 


1.79 

23 

i 

i 


1.00 


t 1 
1 1 





24 

i 

i 


1.00 


1 1 
t t 

1.79 




25 

i 

! 


1.00 


I t 
t t 

-3-35 

1.28 


-1.38 

27 

1 

1 

_J 


1.00 


1 t 
1 t 
f t 

-6.23 

1.33 


-5.55 

13 

1 

1 

! 



1.00 

| {2.46 

1.09 


1.23 

-1.79 

14 

1 

1 



1.00 

1 11.91 

6.71 


1.39 

3.19 

26 

1 

1 



1.00 

1 t 
t 1 


-1.15 


1.82 

28 

1 

t 



1.00 

I 1 
t t 

1.62 

1.12 


3.02 

29 

J 

! 



1 . 00 

t I 
I t 





30 

1 

! 

J 



1.00 

t 1 
t t 
-t-f 

-1.97 



-3.42 

10* 

1 

t 

! 




i.oo ; ; 

4.91 



5.22 

16 

! 

1 




i.ooj ; 

4.44 



3.55 

18 

1 

1 

f 

_1 




i.oo j | 1.07 

1 1 

l_! 

2.60 



2.40 


a. The numbers in Italics had the highest loadings. 

b. Only those loadings with an absolute value of 1.00 or greater are 
shown . 

Co The items which are starred (*) approximate the target 0 
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TABLE 4 


PROCRUSTES ROTATION OF THE AD VAN Cl 
CLASSIFICATION OF RIGHT ANSWERS 




Correlation 

Between Primary 

Axes 



I 

ii 

hi 

IV 

V 

1 

1.00 





11 

- . 96 

1.00 




111 

-»93 

.04 

1.00 



IV 

.80 

1 — i 
cm 

. 

i 

-.83 

1 „ 00 


V 

.95 

-1.00 

-.94 

o 90 

1.00 


The primary axes (Table 4) were highly correlated, suggesting 
that by this classification system, there may be only one factor 
present. 

As indicated on page 88 an identical procedure was used to 
examine the data for the possible presence of "content” factors. 

Table 5 (see: p* 62) gives the target matrix and the pattern on primary 

for this Procrustes rotation solution. 

The fit of this matrix to the target based on content is only 
slightly better than for process-oriented advance classification. 

Factor V loading with items 21, 24, and 25 show a nearly simple 
structure which coincides with the target matrix*. These three items 
also formed a unique cluster on the basis of the cluster analysis 
conducted later in the study. It is possible, however, to give a 
process interpretation to the cluster which may mean that this pattern 
for content might be coincidental. 

Aside from these items the pattern did not reproduce the target 


TABLE 5 


PROCRUSTES ROTATION OF THE INFORMATION 
CONTENT OF RIGHT ANSWERS 


Item No. 



Target Matrix 


! 

f 

Pa 

Item on 

primary 



Stupidity Awareness 

Agression Discipline 

Progress J 

! 

! I 

II 

III 

IV 

V 

i 

loOO 




1 -53 b 


.60 a 



2 

n 

1.00 




! -55 




.56 

3* 

1.00 




; .7? 

06I 




4* 

loOO 




; .69 





5 


1.00 



f 

1.13 


1 0 24 


6* 


1.00 



i .76 


.67 



7 


loOO 



! 

! 

o52 

.51 

.63 


8 



loOO 


Ji-05 


08I 

065 


9 



loOO 


I 

! 

-069 


-1.01 


10* 



1.00 


! 

f 


°53 



11 



loOO 


I 

! 





12* 



loOO 

, 

» 

t 

.67 




13 

.70 


.70 


11.10 

o9i 

0 66 

.66 


14 

.70 

o 

o- 

o 



! 065 

1.30 

o57 

I.05 


13* 



loOO 


! 

t 

o78 

-.50 

1 0 43 


16 



loOO 


t 

1 


-lo04 



17 



loOO 


l 1 K 

<>73 

o?8 

069 


18* 



1„00 


! 

t 

.48 

-o77 

1,21 


19 




o 

o 

1 — 1 

J c.54 

1.13 


1,08 


20 




loOO ; 

1 

! 


^4 



21* 




i„oo ; 

! 

t 




.73 

22 




loCO J 

! 

1 



^2 


23 




io00 ; 

: .47 


.47 



24* 




loOO ! 

! 

1 

. 64 



0 66 

25* 




o 

o 

1 — 1 

I 

! 




.63 

26 




i.oo ; 

1 





27 




i.oo | 

! -53 





28 

.50 

.50 

.50 

.50 ; 

! -85 


.71 

069 


29 



• 70 

o70 J 

! 

1 





30 

. 44 

. 44 

.44 .44 

o44 | 

! 

f 






a. The numbers in I tal i c s had the highest loading's. 

b. Only those loadings with an absolute value of .44 or greater are shown. 

. v?l * ] • *h arc ct arred ( *) approxi n a • o the target . 


matrix in an acceptable manner. Content would 
better than process an a means of classifying i. 


^3 

em to be only slightly 
ms in advance of their 


use . 


Table 6 shows the correlations between the primaries. 


TABLE 6 

PROCRUSTES ROTATION OR THE INFORMATION 
CONTENT OF RIGHT ANSWERS 


Correlation between Primary Axes 



I 

II 

III 

IV 

1 

1.00 




II 

.14 

1.00 



III 

-.76 

-.39 

1.00 


IV 

-.52 

-.81 

.58 

1.00 

V 

.30 

-.12 

- . 46 

-.08 


The In t e rpre t a t i on of Right Answers Using Cluster Analysis 

The negative results just reported suggested the need to search 
for a multiple interpretation possibility 0 Hence, a cluster analysis 
procedure which normalized the same factor matrix by rows as was used 
in the two solutions just given. The normalization involves dividing 
each member of a row in the factor matrix by the square root of the 
communality. Since this value is the length of the vector given by the 


th i s di vi s :i on rai ses 


row of factor loadings, 


this length to unity (one) « 


The procedure then calculates the interpoint distances from the 
ends of the vector pairs, surface-to-surface, across the hypersphere e 
The square of this distance is the sum of the squares of the differences 
across the rows taken in pairs*, This interpoint distance is then used 
to form clusters in which the wi thin-cluster distances are minimised as 
indicated by the formulae given in Table 4l , p* 151- • The clustering 
begins with as many clusters as variables and ends with all variables in 
one cluster*, In addition, there is a unique cluster solution for each 
factor solution which might be used or with the inclusion of each 
additional factor 0 The experimenter, therefore, was left with the 
problem of determining which of many possible solutions to choose. 
Repeated attempts suggested an advantage to the process classification 
of Bloom 1 s Taxonomy o It was decided, to consider a cluster to have 
recapitulated the advance classification if it contained more members 
from one particular advance category than from any other category*, The 
solution which gave the best recapitulation was then sought, by itera- 
tion, for both right and wrong answer clusters. The cluster was 
assumed to be identified on the basis of the recapitulated category. 

For the right answers, in this sense, the best solution was 
derived from an un rotated principal axis factor matrix of six factors 0 
In this solution twelve of the thirty items recapitulated in the 
clusters. This result is four times better than the Procrustes rotation 
to fit content just reported. Since this was 40 per cent of the items, 
in spite of multiple interpretation possibilities, the result was 
reasonably sat i s f ac tory . 


An examination of the data suggested that the first im rotated 
factor might be a '’difficulty 1 ’ factor 0 Table 7 on page 66 expands upon 
this relationship* 

In general, the value of the loading on Factor 1 seems to be 
about 50 per cent of the value of the difficulty 0 The correlation 
between these two variables was r = .65o 

Does this finding seriously disrupt the use of the six factor 
solution? Table 8 on this page compares the interpoint distance 
clusters as determined with and without Factor 1. 


TABLE 8 

RIGHT ANSWER CLUSTERS 
WITH AND WITHOUT 
FACTOR 1 




With 

Factor 1 


? f 
l ! 

I ! 
f .! . 



Without Factor 1 

C 1 

1 

o 

8 

28 

1 ! 
I 1 

i 

2 

8 

22 a 17 

C 2 

3 

17 

30 


! 1 
S t 
! ! 

3 

30 



c 3 

4 

13 

6 


1 ! 
f 1 
f I 

4 

6 

13 


C 4 

5 

14 

19 


1 ! 
t ! 
1 1 

5 

14 

19 


C 5 

7 

22 

23 


t 1 
! ! 
t 1 





C 6 

9 

27 



! 1 
1 f 
1 1 

9 

27 



C 9 

10 

11 

16 

29 

1 t 
1 » 
{ | 

10 

29 

11 

16 26 

7 





t t 






C 8 

12 

20 

26 


! f 
t ? 

f f 

7 

12 

20 


C 9 

15 

18 



1 ! 
1 ! 





c in 

21 

25 

24 


1 ! 
1 ! 
f i 

15 

18 

24 

21 25 

1 u 





i i 
. -j-i - 






a,c The numbers in Italics displaced with the removal of Factor 1. 


1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 

21 

22 

23 

24 

25 

26 

27 

28 

29 

30 


Co 


TABLE 7 

RELATIONSHIP BETWEEN ITEM CONSISTENCY, ITEM 
DIFFICULTY AND ITEM FACTOR LOADING ON 
UNROTATED FACTOR 1 


Internal 

Consistency Difficulty Factor 1 


.384 

. 086 

-.007 

.456 

. 245 

.150 

.304 

.345 

.176 

.425 

.173 

.037 

.295 

.223 

- . 200 

.473 

.525 

.392 

.336 

.144 

-.185 

.419 

.295 

.022 

.135 

. 698 

.245 

O 

ro 

o 

ol73 

.074 

.334 

.475 

. 216 

.430 

. 345 

.181 

.378 

. 669 

.118 

.405 

.460 

.114 

.015 

.050 

-.581 

.447 

.813 

.314 

.546 

.079 

.194 

• 295 

.014 

-.732 

. 463 

.309 

.055 

.269 

.532 

.055 


. 404 

.324 

O 

O 

0^ 

0 

.001 

.072 

-.587 

.165 

.583 

-.084 

.377 

.8 56 

.402 

.329 

.547 

.340 

.140 

.381 

.101 

.338 

.849 

.414 

.378 

.367 

.048 

. 275 

.827 

.243 

♦ 

M 

V-T\ 

OO 

.734 

.182 


The Italics 


67 

in Table 8 on page 65 indicate that seven items move 


by dropping Factor 1. This fact does not seriously affect the replica- 
tion of the advance classification in the new solution,, Also, most of 
the items which move to a new cluster do not follow the general rale 
that the difficulty be roughly twice the factor loading. For these 
reasons, it was decided to retain the complete six factor solution 
throughout subsequent analyses 0 

Since a Procrustes rotation was used to determine how well the 
advance classifications fit the data, it was reasonable to use the same 
procedure with the cluster analysis data D Table 9 (see: p Q 68) reports 

the target and pattern matrices in solution,, 

The pattern on primary in Table 9 is a good reproduction of the 
simple structure of the target matrix 0 Furthermore, if the values of 
M h ,? (i„e. the square root of the communal ity from the six factor matrix) 
are taken into account, then the pattern seemed to fit even better G The 
cluster analysis was produced from the interpoint distances derived from 
a normalized matrix,. For this reason, to find the length of the largest 
vector approximating the overall length of the vector in the unrotated 
six factor solution add to the acceptability of the solution 0 


In short, the cluster solution was a far better fit to the data 
than either of the two methods of advance classification,. The independ- 
ence of the interpoint distance clusters from rotation problems also 
reinforces the acceptability of this approach. Table 10 (see: p. 69 ) 

reports the correlation between the axes. 

The relatively low correlations between the axes in Table 10 
further strengthens the support for the use of interpoint distances as 
an analytical technique for trie problem under examination in this study „ 


TABLE 9 


PROCRUSTES ROTATION OF THE INTERPRETABLE CLUSTERS OF THE RIGHT ANSWERS 


Item 



Target Matrix 




1 t 
! ! 
! t 


Pattern on Primary 



No. 

,a 

h 

r* 

a 

C 5 

<L 

0 

CL 

< 

C 8 

C 10 

11 c 1 

S 

n n 

0 V 

C 8 

C 1C 

1 

.40 

1.00 






| J . 40 b 





2 

• 55 

1.00 






jj.43 





8 

.62 

loOO 






11.63 





28 

.71 

1.00 






j | p 6 2 





7 

<>49 


1.00 





t ! 
i i 
t ! 

,52 




22 

.71 


1,00 





! I 
T J 

.77 




23 

.51 


1.00 





! ! 
1 ! 
1 ! 




-.43 

9 

*52 



1.00 




: ! 
! ! 
! t 


.48 



27 

• 71 



1.00 




t t 
l f 
! I 


.49 



10 

• 57 




1.00 



i t 

: t 


,44 



11 

.48 




1.00 



! f 
t ! 


.46 



16 

29 

.60 

.28 




1.00 

1.00 



! ! 
! 1 
1 ! 
1 1 


49 

.23 



12 

.70 





1.00 


t ! 
! I 
f I 



.68 


20 

.57 





1,00 


f ! 
t ! 



.46 


26 

.61 





1.00 


! t 
1 ! 
! I 



.40 


21 

.61 






1.00 

t ! 
! f 
! 1 




. cO 

2^ 

.68 






1,00 

t t 
! I 

- 0 A 9 



• 58 

25 

.62 

' 





1,00 

I ! 




o5^ 


a. The symbol M h M stands for the square root of the communality. 

b. Only those loadings greater than an absolute value of .35 are reported. 

c. This loading, although /o35? is reported because of its low value for ’h 1 . 


TABLE 10 


69 


PROCRUSTES ROTATION OP THE 
CLUSTER ANALYSIS OP THE 
RIGHT ANSWERS 



Correls 

it ion 

between 

primary 

axes 



n 

5 

r 

C 7 

. 

°8 

C 10 

.00 






.31 

1.00 





O 

O 

.18 

1.00 




.17 

1 — 1 
CX 

1 

0 

vx> 

1.00 



.06 

-.29 

ox 

0 

1 

-.08 

1.00 


.12 

.20 

cl5 

-.12 

.09 

1.00 


The largest correlation between axes given on Table 10 was -.31 which 
represents an angle of more than 70° between this pair of axes. It is 
possible, therefore, to state that interpoint distance clusters produced 
a solution which was independent of the usual rotation problems and 
which was approximately orthogonal. The procedure gave a very satisfac- 
tory statistical representation of the data. On the other hand, the 
failure of the advance classification systems to render interpre labil- 
ity left the researcher with the problem of interpreting these clusters 0 
Finally, a matrix which is in fact categorical in the sense 
given on page A 7 would be expected tc have low correlations between the 
primary axes for a Procrustes rotation. When the dimensionality of this 
categorical matrix has been reduced by factor analytic techniques, 
before rotation, the best fit from a Procrustes rotation should display 
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the retained lengths of the vectors (i 0 e 0 the square roots of the 
communal i ties) as the loadings of the pattern on the primary axes Q 
Also, the pattern on the primary axes should display a structure similar 
to the target matrix 0 All three of these properties must be present for 
the inference to "be made that the principal axis matrix is categorical o 
It is evident from Tables 9 and 10 (pages 68 and 6 ^) that all these 
conditions were met, strongly suggesting that the cluster analysis 
technique gives a good categorical solution to the six factor data 0 The 
low internal consistency was thus explained, suggesting profile scores 
from the clusters may better describe the data than total correct scores 
alone c Finally, the close match between the original vector lengths and 
the facdor loading in the Procrustes rotation and the near orthogonal: ty 
of the factors suggests that this solution clearly identifies the homog- 
eneous subtests of the right answers c These findings contradict the 
apparently poor showing of this test in the usual analytical settingo 

The Meaningful. I n t e rp re t a t i on of Item Clusters 

With the failure of the advance classification, the multiple 
interpretation hypothesis was the only alternative to investigate hence 
some reasonable common ground was needed for each cluster if answering 
was systematic. The first possibility which had to be either confirmed 
or eliminated was that the clusters were sufficiently strongly content 
oriented to suggest content as a possibility. The only cluster in which 
content was at least a strong contender to process was the Cluster C_^ 
containing items 21, 24, and 25o All these items were based upon reading 
selection number five (Progress), but also, all three were classified as 
Analysis items. A closer look was talcen at this cluster, along with all 
the others (see: Appendix C) 0 This look supported the process 


Thus, it was reasonable 


interpretations over the content interpretation,, 
to reject content as a basis for the interpretation of any of the right 
answer clusters 0 

The misfit items in the identified clusters were examined by 
logi co -semantic (structure-meaning) analysis to determine if they could 
be reasonably reclassified in common with the recapitulating category 0 
Success by this procedure would lend support to the multiple interpreta- 
tion hypothesise All misfit items in clusters C., , except possibly 

item 7 could be reclassified to fit the overall classification of these 
clusters, adding three or four items to the original twelve,, 

The logico-semanti c analysis of Cluster 0 o revealed that the 

o 

formation of an inductive structure within a particular reading selec- 
tion could possibly he the basis for synthesis items. For this reason 

C D was classified as Synthesis, addins another three items to the 
o 

support of the multiple interpretation possibility «> 

This procedure gave 19 of the 30 items, or 6 3 per cent, of the 
items a reasonable classification based upon process 0 Since the inter- 
rater reliability was only moderately high (r - c83) and since the cross- 
validation based upon interpretable reasons found in Powell ( 1968 ) was 
only 64 per cent, this level of recapitulation can be considered to be 
satisfactory o 


For the remaining four clusters the Interpretation was ambiguous 0 
Cluster containing items 15 and 18 seemed to be simply poor items 0 
Cluster C^ seemed to involve implication or extrapolation from the rel- 
evant content suggesting comprehension but in the absence of better 
definitions for strategies it was safer to call this cluster ambiguous 
(see: p. 197). The remaining two clusters (c and C^) seemed to be 
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strongly influenced by the nature of the foils in the items which tended, 
in general, to lower them to Comprehension items; but each had some 
Analysis characteristics leaving their classi fication ambiguous, which 
seemed to further support the multiple interpretation hypothesise These 
decisions are summarized on Table 11, page 73. 

Trying to make Synthesis items by combining two or more reading 
selections proved unsuccessful for a number of reasons, suggesting the 
need for more research on this method*, 

Thus, the results of the logico- semantic analysis of the item 
clusters suggested reasonable support for l) the multiple interpretation 
hypothesis, 2) the transcendence of process over content, and 3) the 
suggestion that foils influence item performance *> 

The summary just presented is supported by a detailed discussion 
given in Appendix C (see; p c 186 ff) . The details are also presented, 
first, because it was felt that the effective reclassification repre- 
sented evidential support for the multiple interpretation hypothesis, 
and second, because subsequent researchers might find value in an 
independent evaluation of the logic behind the conclusion of this study*, 


The Meaningful Interpretation of Wrong Answer Clusters 

The tables for the factor analysis of the wrong answers were 
very large since they involved 90 variables (see: Appendix A, Tables 3^ 

to 39). 6 The analysis of the right answers showed that interpoint 
distance cluster analysis gave a good representation, in a statistical 


The phi coefficients in these tables are in the following 
sequence: variables 1 to 30 represent II) to 30 b-,; variables 
31 to 60 represent 1D 0 to 30D o ; and variables 6l to 90 
represent ]D^ to 301^3 


TABLE 11 


CLASSIFICATION AND MEMBERSHIP OF 
RIGHT ANSWER CLUSTERS AS 
DERIVED FOR GROUP A 


Cluster 

Label 

Item 

Membership 


Advance 

Ciassif i cation 

Interpreted 
Ciassif i cation 

i — 1 

o 

pa 

2 

8 

28 

Analysis 

Analysis 

C 2 

3 

17 

30 



Ambiguous 

C„ 

;> 

4 

13 

6 



Ambiguous 


3 

14 

13 




C 5 

n 

i 

22 

23 


Analysis 

Analysis 

C 6 

9 

2? 



Analysis 

Analysis 

C ? 

10 

11 

16 

29 

Evaluation 

Evaluation 

CO 

o 

12 

20 

26 



Synthesis 

°9 

15 

18 




Ambiguous 

o 
. — 1 
o 

21 

25 


24 

Analysis 

Analysis 


a. The numbers in Ital 1 cs were all in the category named in the 
advance classification. 

'bo The "interpreted classification" of each cluster is given in 
Appendix C beginning on page 186. 
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sense, for those data. Finally, the inter.* -a ter rel :i ability was Lowe - 
for wrong answers than right answers, suggesting that the advance 
classification of foils would be less likely to reappear in the data 
than the right, answer. For these three reasons, the interpretation of 
the wrong answers began with the cluster analysis. Attempts to fit the 
unrotated principal axis factor matrix to the advance foil classification 
and to the foil content were not made. There was no reason on the basis 
of the characteristics of the results of the cluster analysis to assume 
that the results of these two preliminary steps would have been 
substantially different for the wrong answers than they were for the 
right answers. 

The best replication of the advance foil classification which 
could be found in the cluster analysis involved a 25-factor solution to 
the phi correlation matrix of the wrong responses, and 15 clusters of 
foil in this solution,, This replication placed 28 of the 90 foils 
(or 31 per cent) into clusters which might be considered equivalent to 
the categories of the advance classification on the basis cf the most 
frequently occuring category of foil in that cluster. This proportion 
(31 per cent) v/as not quite as good for the wrong answers as the 
corresponding proportion (40 per cent) was for the right answers. 

In getting this best replication, the same procedure was used 
for determining the number of clusters as was used for right answers. 

All 90 foils were clustered. Once the best replication was 
found, however, those foils for which the selection ratio was less than 
.06 were dropped from further consideration. This precedent was 
established when the interpretation of the right answer clusters was • 
being made® The result cf the dropping cf foils cf low selection ratio 
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was to reduce the number of foils vender considers, won from JG to 60* 

Of these 60 foils, 18 foils (or 30 per cent) continued to meet replica- 
tion requirements. The proportion is essentially the same as before 0 

The interrater reliability for foil classification (r = .62) 
was not as high as for the right answers. It is possible that this 
figure might have been considerably lower, had the right answers to the 
items not been clearly indicated to the raters at the time of the rating. 
The procedure of rating involved comparing the foil, stem, and right 
answer relationships to the definitions of foil categories as listed 'in 
Chapter III. It became fairly evident that at least some of the foils 
might be placed quite reasonably into several different categories. The 
problem of multiple classification of foils will be dealt with in more 
detail in the interpretation of the wrong answer clusters. 

(sees Appendix C, p. 209 ff). 

Briefly, the advance classification of foils were arranged into 
three or four possible general classes. These general classes were 
l) Strategy Errors, 2) Misreading, 3) Misinformation, and A) 0ther o 
The Other, (o), category was for foils which for some reason could not 
be readily classified into some established category. In the Experi- 
mental Test this category referred primarily to the foils for items 19 
to 24 inclusive. In these items a different item format was used to 
that of the remainder of the items. It was not possible, in advance, 
to know whether or not this difference in item format would influence 
the way in which the foils behaved statistically. It was assumed in the 
interpretation of the clusters that M O n type foils which were found to 
be in reasonable association with foils of specified categories were not 
influenced in their behavior by the item format. If the Other (o) type 


foils formed their own unique clusters, these clusters were assumed to 
represent categories of foil not identified by the advance classifi- 


cation of foils. Two such categories appeared in the data. 

The specific categories used in the advance classification of 
foils were as follows: 



Name 

Symbol 

lo 

Over general izat i on 

OG 

2. 

Overs implj f icati on 

OS 

3° 

Substitution 

Sub 

4. 

Inversion 

Inv 

5- 

Invalid Assumption 

IA 

6. 

Irrelevancy 

Irr 

7o 

Common Kis concept ion 

CM 

6. 

Word - Word L ink 

WW 

9. 

Transposition 

Tr 

O 

i — ! 

Redefinition of Terms 

RT 

i — 1 

i — l 

Other 

0 


A cluster was identified on the basis of the most frequently 
occurring foil of a particular advance classification in that cluster. 
The identification given in Table 12 (see: p c 77) is based on all. of 
the foils in each cluster before low selection ratio foils were 
eliminated. This identification was used as a starting point for the 
attempt at meaningful interpretation of each foil cluster 0 The final 
meaningful interpretation of each cluster is also given in Table 12. 

Once again the members of each cluster were examined in an 
attempt to determine the common basis upon which these foils clustered 
together 0 Particular attention was paid to the foils which were not in 


TABLE 12 


CLASSIFICATION AND MEMBERSHIP OF WRONG ANSWER CLUSTERS AS DERIVED FOR GROUP A 


Wrong 




Foils in 

Each 




I den t if i ca t i on 

Interpretation 

Answer 




Wrong 

Answer 

Cluster 



by Advance 

by Logi co-semantic 

Cluster 










Foil Classification 

Analysis 

w i 

1D 1 

2D 1 

23D 1 

221> 3 

8D 

17D 1 




OG 

OG 

W n 

q-p a 
^i 

20D 1 


24D 1 

4D. ? 

18D q 

j 

~2 

23D 3 

23D-, 

j 

29D 3 Sub 

Eliminated 

W„ 

•) 

4Di 

26 D 1 

30D 1 

<D 

2 

6D 0 

2 

14D 3 

13D ? 

29 D 2 

21D 2 

T T b 

vJ 

CM 

W 4 



ud 3 

27D X 

1D 3 

30D 3 




u 

RT 

W < 

6D 

16P 

12D 0 

hd 2 

12D 2 

2D. 

19D 2 

20 D 3 


0 

Inv 

W 6 

9Di 

17D 2 

25 D 1 

21Dj 

?d 3 

26D 3 




u 

NS 

w, 

( 

10D 1 

19 D q 

4D 3 

13D 3 

12D 

15D 3 

18D 2 

26D ? 

8D 3 

u 

U 

W 8 

11D 1 

28D^ 

13D 1 

14D 2 






IA 

IA 

V 9 

14D 

'—2 

3D 3 

28D 1 

10 d 2 

17D 3 

22D„ 

£ 

5D 3 


OS 

03 . 

ho 

l6p 

—2 

29D 1 







u 

RT 

w 

11 

18D 1 

?I) 2 

6D 3 

16D 3 

2D 2 

10 d 9 

J 




WW 

WW 

W 12 

19D ] _ 

22D 1 

2H 3 

24D 9 






0 


1 — 1 

21D 1 

_L 

28D 2 

23d 2 







0 

0, 

W 

14 

3D 2 

25B 2 

9D 2 

27D 2 






Irr 

Irr 

W 

15 

i^d 2 

30E 2 

24D 3 

27D 3 






U 

Tr 


a„ Foils indicated by Italics were dropped from the analysis on the basis of a low selection ratio after 
the factor analysis was completed „ 


o 


be The U stands for an "unelassif ied " cluster 
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the common advance category present in the cluster© In addition, the 
evident influence of foils upon the performance of an item led to the 
possibility that the interpretation of a foil by category might depend 
upon the point of view of the interpreter© The 'Low interrater reli- 
ability suggested this possibility© For at least some foils several 
classifications may have been reasonable© In this case the foil cat- 
egories would not be independent or unambiguous© If foils are only 
interpretable after a test has been given, then these interpretations 
become specific to the particular examinees upon whose performance the 
interpretations were based. That is, the expectation would be that 
where multidefined foils were concerned, the proportion of cross- 
validating foils on a cluster- for-clustcr basis would be low© 

A procedure similar to the one employed for item clusters was 
used with wrong answer clusters. For complete details see Appendix 0 
(see : p. 209 f f) . 

The possibility of multiple interpretations of wrong answers was 
most clearly illustrated in the case of foil 2TL . Briefly, the class! fi 
cation of this foil was initially set as Substitution (see: p© 156) 

because it substituted a conjunctive for a disjunctive relationship© 
However, it is possible to look at these relationships as Ove rgener aliza 
tion or Oversimplification as the discussion in Appendix C indicates 
(see: p. 213 ). 

Apparently, considerably closer analysis of the logico-semantic 
relationships within items and their components would probably reveal a 
range of possible interpretations for each item© This multiple 
interpretation effect is consistent with the findings using right 
answers© For foil clusters, the lower recapitulation left five clusters 
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containing 28 foils from the 60 that remained classifiable because of 
the advance classification of their members 0 Of these, ten foils did 
not have these classifications in advance, but eight of them could be 
reasonably reclassified into the category of the total cluster « 

Only one cluster (w ) which contained nine foils could not be 
classified by logico-semantic analysis although one or two more of these 
were shaky at best 0 One cluster (w^) was classified Common Misconception 
on the basis of logico-semantic analysis because the characteristics of 
this cluster were reminiscent of the findings of Powell and Isbister 
( 1969 ) thereby suggesting consistency between two independent groups on 
the sajne test c Three categories (MS, 0^ , 0 o ) were added to the Guide- 
lines because of the clustering as well as the logico-semantic analysis 0 
Clearly, the Guidelines were not exhaustive and were not mutually 
exclusive, as already anticipated by the establishment of the "Other 1 ' 
classification, and by the multiple interpretation hypothesis 0 Because 
of the support for this hypothesis it is likely that the interpretations 
formulated should be confined to the group upon which they were derivedo 

A Possible Hierarchy of Foil Categ ories 

During the above analysis it became evident that foil categories 
were, to a degree, interchangeable . For instance, foil 2D^ could be 
classified as 0G, OS, or Sub Q In only a few cases did reclassification 
within a wrong answer cluster of specific foils seem unreasonable. No 
attempt was made to exhaustively reclassify foils. Instead, the attempt 
was to reclassify specific foils to fit the pattern which seemed to be 
evident within the entire cluster as derived from Group A. Table 13 on 
page 80 summarizes the reclassification which occurred for each foil 
class in each of the wrong answer clusters 0 


30 


TABLE 1.3 


RECLASSIFICATIONS WHICH V/ERE 
MADE OF SPECIFIC FOILS 



Cluster 

Advance Cl a s s i f i ca t ion 

R e classification 


w i 

Sub 

became 

OG 

w 

Sub, OS and OG 

became 

CM 

W 6 

OS and OG 

became 

Inv 

Inv, Sub, OS, Irr 

became 

NS 

W 8 

Inv and Sub 

became 

IA 

W 9 

W 

11 

Irr, Inv, and Sub 

became 

OS 

Irr and OS, Sub, CM 

became 

ww 

W 13 

OG 

became 

O 

ro 

i 


Since the 0 class in Table 33 was treated essentially as though 
it were unclassified it was omitted from this table, as were foils which 
did not classify within their respective cluster 0 Table 13 lists the 
Advance Classification of the foils in each of the 'wrong answer clusters 
which were different from the final classification of that cluster and 
the final classification given c As such it summarizes Tables 53 t° ^3 
inclusive „ 

Put the other way around, the substitution category of foils 
disappeared by reclassification as OG, CM. NS, and IA. Besides 


retaining a cluster for itself, 01 also reclassified as CM, Inv, and 0 . 
OS also retained its own category but also because VW, CM, Inv, and NS. 
Similarly, some of the Inv foils reclassified to NS, IA, and OS. 

Finally, some Irr foils became NS, OS, and WV/ 0 Figure 2 presents these 
changes diagrammatieally . 



FIGURE 2 

FOIL RECLASSIFICATION PATTERN 


In Figure 2 there is a double headed arrow between OS (Over- 
simplification) and Inv (inversion). This arrow means that at least one 
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OS foil was reclassified as an Inv. and at least one Inv foil was 
reclassified as an 03 (Oversiinpli.f ioaticn) 0 The other arrows in the 
figure can be interpreted in the same way. 

Sub foils disappeared, and hence are dropped from this diagram. 
From the analysis of the right answer clusters it became evident that 
CM and WW type foils tended to lower the level in the taxonomy of an 
item while OS and 00 foils tended to raise it. 

From the reclassification pattern in Figure 2 OS foils seem to 
be pivotal in the sense that it was most frequently involved in changes 
(from or to). Furthermore, NS, Inv. and 00 may be higher order foils, 
and CM, WW, and Irr of lower order foils on the basis of the changes 
these foils seem to cause in the reclassification of right answers . 

Thus the general order of a possible foil hierarchy would seem 
to foil ow a vertical axis in Figure 2, with the lowest level at the 
bottom. 

0^ and IA foils are ambiguous in this pattern because they have 
one-way linkages only. 

Other evidence to support the possibility of an hierarchy of 
foils should be found before this property of foils can he considered to 
be established from the data 0 Any set of .random variates can be ordered 
on the basis of relative magnitude into an arbitrary hierarchy. Two 
random variables will be uncorrelated. If, however, two correlated 
hierarchies can be produced from two independent variables, this produc- 
tion would suggest that these two variables may be functionally related. 

One possible source of such an arbitrary hierarchy is to consider 
the average total-correct score of the respondents selecting each inter- 
preted foil as listed in the Appendix in Table 40 „ The average of these 
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averages could he found for each interpreted wrong-answer cluster 0 This 
average total-correct score for all the members of a particular cluster 
may reflect a systematic characteristic of the examinees which may 
relate the kind of "error" made to the total-correct score achievedo 

If foil selection patterns reflect a functional relationship 
between "errors" and total-correct score, the rank cf a foil (from one to 
three for high to low) within an item based on the average total-correct 
score should also reflect this functional relationship® If each item is 
independent of each other item, the average within item rank of the foils 
across a cluster should also be independent of the average total-correct 
scores of the foils in that same cluster G This is a reasonable assump- 
tion since only in item cluster does there not seem to be a close 
relationship between item clusters and foil clusters for the items in 
that cluster. An exception to this expectation would arise in the event 
that there is, in fact, a functional relationship between the kind of 
error and the total-correct score 0 In this latter case, both these 
procedures should produce roughly the same hierarchy. 

In addition, this hierarchy would be expected to reflect the 
pattern which seemed to be evident on the basis of the pattern of re- 
classification of foils, the reclassification of items as influenced by 
foils, and the indications which also arise from other research into 
wrong-answer patterns 0 

Since this part of the study is exploring the possibility of a 
hierarchy among the foil categories, the rank order of the two variables 
just described was determined. That is, the rank order of the average 
for each interpreted foil cluster of the average total-correct score 
associated with each foil in that cluster was found 0 This procedure 
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established an arbitrary ordinal relationship among the interpreted foil 
clusters. A ranking of the average within item ranks was also estab- 
lished. If there is no functional relationship between foil type and 
total-correct score, the rank order correlation between these two 
ranking systems should be near zero. That is, the within item and 
between item characteristics would not be related to the average total- 
correct score on each foil in a functional manner. Table IT shows the 
ranking of foil clusters by two independent methods (see: p* 85). 

Each of the two ranking systems in Table l4 tend to support the 
more general ranking procedure suggested by the reclassif ication pattern. 
The comparison between the two ranking systems gave a rank order correla- 
tion of p = .68 which is significant (p - .01 for two-tailed test for 
N = 12). 


Other findings support this hierarchy. For instance, Powell and 
Isbister (1969) found that IA foils correlated negatively to Synthesis 
items. Foils in this category would be expected to have the fairly low 
rank this study suggests for IA foils. On the other hand, placing this 
foil type just above the content-linked misreading categories may be 
placing it too low in the hierarchy. Similarly, RT foils seemed to be 
content-linked, and would be expected to have a low rank for this reason. 
Support for this extrapolation was also evident. The tendency for Irr 
foils to distract middle and high level performers has already been noted 
in Powell (1968) and Powell and Isbister ( 1969 ). Thus the ranking by 
average total-correct score which seems to place Irr foils in sixth 
place would seem to be too low. Similarly, the ranking on a within item 
basis as sharing top place would seem to be too high. 

By itself, the ranking on total-correct sc 


ores would seem to be 


85 


TABLE l'l 

RANKING OF FOIL CLUSTERS BY TWO 
INDEPENDENT METHODS 


Interpreted 

Rank by total 

-correct 

Rank by average 

foil 

cousters 

averages w 

;i thin 

within item 

rank 



clus ter 

s 

by clusti 

er a 

Cluster 

In t e rp ret a t i on 

Average 

Rank 

Average 

Rank 

W 1 

OG 

12.0 

1.0 

2.4 

1.5 

w 13 

°2 

11.9 

2.0 

2 . 6 ? 

4.5 

W 6 

NS 

11.8 

3.0 

2.6 

3.0 

w 5 

Inv 

11.7 

4.0 

2.7 

6 . 0 

w 

9 

OS 

11.6 

3.0 

3.0 

9-5 


Irr 

11.5 

6.0 

2.4 

1.5 

w 

15 

Tr 

11.3 

7.0 

2.75 

7.0 

V 

3 

CM 

11.2 

8.5 

2.9 

8.0 

W 8 

I A 

11.2 

8.5 

o 

1 

4.5 

w 4 & W 10 

RT 

11.1 

10.0 

3.25 

i — l 

i — 1 

W 11 

WW 

11.0 

11.0 

3.0 

9-5 

W 12 

°1 

10.8 

12.0 

3.25 

'-T\ 

i — 1 
i — 1 


a. The within item rank includes the right answer and does not drop 
any foils. 


fairly closely related to an hypothetical foil hierarchy on a between 
item basis. Similarly, the within item ranking would seem to be more 
closely related to the influence of foil categories upon the items. 
However, these two variables are obviously related, as indicated by the 
significant rank order correlation between them., These results suggest 
overall systematic answering which influences the statistical outcomes 
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of both within and between item events. 

Perhaps the most reasonable arrangement for the hierarchy would 
be to consider both events to be interdependent. The simplest approach 
in this case, is to consider the average rank of the two separate 
ranking systems used and to rearrange the foil categories accordingly. 
The resulting hierarchy would then reflect the influence of both between 
and and within item events. Table 15 gives the results of this 
procedure . 


TABLE 15 


RERAN XING OF FOILS 


BY AVERAGE RANK 


Cluster 


Foil Classification 


New Rank 



11.0 


12.0 


3.0 

4.0 
5-0 

6.0 
7.5 
7-5 
9.0 

10.0 


1.0 


2.0 
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This rearrangement put RT nearer the hot tom, OS nearer the 
middle (as its pivotal position suggested), and Irr nearer the top than 
the average total-correct rank, which seems to be reasonable relative tc 
the available evidence. This new reordering has not- been used in .sub- 
sequent analysis because of the problem of the relevance of the within 
item ranking to this hierarchy. Further research is needed before the 
most probable sequence in the hierarchy has been established. 

The lowering of the performance level of synthesis items in 
and C 0 by Tr, CM, and RT foils further supports this hierarchy. Foil 
17D 3 was reclassified as OS in the logico- semantic analysis of the wrong 
answer cluster. The pivotal position of the OS category would seem to 
support the M double-strategy n interpretation of C^. Foil was 
changed from OG to CM, which suggests that this foil, and Tib which was 


unclassifiable may have combined to low r er this analysis item to a 
comprehension level. Right answer cluster C 0 may, therefore, be a 
comprehension level cluster and not a double- strategy cluster as 
suggested earlier. These suggestions are too tentative to alter its 
"undetermined " classification . 

Further support for this foil hierarchy may be found by taking 
the new foil ranks and determining from these the average foil rank of 
each right answer ciuster 0 Table 16 (see: p. 88) gives this 

information. 

Table l6 shows three additional characteristics which may add 
support to the concept of a foil hierarchy. First, the order conforms 
to Bloom ! s Taxonomy with C^, the apparent split strategy cluster falling 
between the analysis and undetermined categories. Second, evaluation 


fell out of order as occurred in the Kropp, Stoker, and Bashaw (l 966 ) 


TABLE 16 

ORDERING OF RIGHT ANSWER CLUSTERS BY AVERAGE FOIL RANK 



Right 

Answer 

Rank 

Right 

Answer 

Cluster 

Interpreted 

Classification 

Foil Types Present 
in Cloisters 

Average 

Foil 

Rank 3, 


1 


Analysis 

OC- (2)"!? OS. 0„, NS. 

WW 

3,8 


5 





2 

n 

n 

Analysis 

OG (3), OS, 0 2 , IA, 

WW 

4.2 

3 

c, 

0 

Analysis 

Irr (2), NS 


5o0 

4 

p 

8 

Synthesis 

Inv (3), NS, CM 


c 9 
J 0 c. 

5 

O 

i — 1 

o 

Analysis 

NS (2), 0 2 , 0,, Irr, 

Tr 

5o5 

6 

c 9 

Undetermined 

OG, OS (2), Irr, Tr, 

RT 

5o7 



(Analysis or Comprehension?) 




7 

C 7 

Evaluation 

OS, IA, Inv, WW (2), 

CM, RT 

8.3 ' 

3 

C 4 

Undetermined 

OS, IA, Inv, CM (2), 

°1 

8o5 

9 

C 3 

Undetermined 

CM (3), WW 


0 i 
y 0 ~ 


( C omp re hen s 1 on ? ) 



10 

c n 

Undetermined 

WW 


11.0 


9 






a Q These figures are the average of the foil ranks by total-correct averages within cluster as given in 
Table 14, page 85 0 

he The number in parentheses indicates the frequency of recurrence of a particular foil classification 
within the right-answer cluster when this frequency is greater than one Q 


oc 
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study o Third, the cluster C c . if legitimately classified by one foil, 
fell at the bottom c The Procrustes rotation match-to-content suggested 
that this cluster might he a content-oriented cluster*, In this case, 
its location was also reasonable *» 

Further support for the hierarchy can be found by determining 
the rank order of the right answer clusters in several ways® When the 
between and within item foil ranks, as given in Table Id, page 85 ? are 
used to calculate the right answer cluster rank, a highly significant 
correlation is found (p = .90, p .001 for N = 9)® This finding 
supports both the hierarchy and the apparent influence of foils upon 
item performance o All of the other possible rankings produced correla- 
tions which were not significant, including a comparison between ranking 
by average foil rank and average total-correct scores 0 

Re sults Relat ed to the Sub sump tive Property o£ the Taxonomy 

Another interesting finding in the results just reported is 
the fact that the average difficulty of each cluster seems to be un- 
correlated with the rank of each cluster*. This correlation is p = o 05 
and does not change much if within item rank is used or if the composite 
rank is used D This finding would seem to contradict the subsumption 
characteristic assumed to be part of Bloom 1 s Taxonomy . 

From the data available in this study, another test of this 
subsumption hypothesis can be made*. If the sub sump tive property holds, 
successively higher members of the right answer hierarchy should be 
obliquely related© The Procrustes rotation gave a good fit to the 
target for the cluster analysis (see: p. 67 )* In general, the 

relationship among the primary axes would seem to be orthogonal © 


However, the sampling distribution of these correlations is unknown so 
that their statistical significance of these correlations cannot be 
determined. In this case, the actual values of these correlations, 
arranged in order on the basis of the hierarchy of right answer clusters 
may reveal a systematic pattern. Table 17 gives this data. 


TABLE 17 

POSSIBLE SYSTEMATIC OBLIQUITY 
BETWEEN ORDERED CLUSTERS 51 



C 7 

1 


i 

O 

rH 

o 

1 

NO 

o 

p p 

c 5 1 

n „ 

-.12 





10 






c h 

<TN 

o 

I 

! 

1 

oTN 

i — 1 



o 


1 

1 



Relations among 

C r 

• 31 

1 

f 

ro 

o 

00 

1 — 1 

• 

Analysis clusters 

5 


! 

1 




i — 1 

o 

-.17 

! 

i 

i 

i 

i 

.12 

! 

o 

i — i 
CN 

1 

C 6 

I 

O 

00 


ON 

o 

1 

rN 

o 

1 

-.29 .06 

1 

t 

f 

— L 


a. This table is based on Table 10, page 6 9 


If the point is stretched to the ultimate and correlations 


greater than or equal to an absolute value of .11 are considered 
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oblique (r ^ j . 1 5 1 ) , there may be seven (7) of these fifteen (15) 
relationships that could possibly be considered as oblique. In this 
case three (3) of these relationships are oblique along the diagonal of 
Table 42, but all are within the analysis grouping. Four of these 
seven are found among the six of the analysis grouping. The other three 
are among the nine relationships outside the analysis grouping. The 
highest of these correlations still represented an angle larger than 70° 
(r = - .31) and the greatest proportion of these slightly nonorthorgonal 
angles are found among analysis clusters. Such slight pattern as might 
have existed did not seem to be too important outside of the analysis 
group of clusters. The relationships among the clusters did not seem to 
support this assumed sub sump tive relationship "between levels of the 
Taxonomy . Kropp et_ al ( 19 66) found that the order from the simplex 
analysis did not consistently support this sub sump tive property either 
(see: p. 21). In short, the lack of correlation between average item 

difficulty and cluster order would seem to be further evidence towards 
the probable refutation of the assumed subsumptive relationship between 
the levels of Bloom's Taxonomy . 

The fact that the correlation between the order of the right 
answer clusters and the average total-correct score was negative 
suggests a general tendency for examinees to do better on low level 
items, which suggests a ceiling effect present in this difficult test. 
The test was also shown tc be difficult by the average total- correct 
score, which was 12.19 out of a possible 30 items for Group A. This 
possibility found further support in the cross-validation part of the 
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C ro s s - val i da t i o n of the Analysis 

The multiple interpretation hypothesis supported thus far, and 
the apparent foil hierarchy, suggested two possibilities with respect to 
cross-validation o First, the attempts made to cross -validate these 
findings might not be successful* Other evidence for systematic 
responses may, therefore, have to be sought, such as the hierarchy, 
which might be supported more strongly in the cross-validation than the 
individual clus te r s . 

The total-correct means an.d variances for each of these two 
groups on the experimental test were used in a t-test for independent 
samples in order to confirm the equivalence cf test scores of these 
two groups o Table 18 contains this data® 


TABLE 18 

COMPARISON BETWEEN GROUP "A" AND GROUP "B" 
ON TOTAL- CORRECT SCORES 


Group A 

Group B 

df 

4 - 

1 / 

a 

P 

X - 12„19 

S 1 2 = 8.62 
\ = 139 

X ? = 11.91 
s 9 2 = 7 o 60 

N ? = 138 

275 

o83 

„40 







a. Probability of t_ equal to or larger than o83 is given for a 
two-tailed test 0 
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On the oasis of the results given in Table 18, page 92, the two 
groups would seem to belong to the same population so far as the 
experimental test means and variances are conce:-Tted 0 

C ro s s - val 1 da t :i on of right answers 0 Tnree different comparisons 
were used in the cross-validation of item clusters 0 l) Those right 
answers from the advance classification which were used to help identify 
the item clusters were checked against the clusters of Group B« 2) The 
clusters which occurred from the answers of Group A were compared with 
the clusters from Group B 0 3) The right answer clusters were divided 
into three groups of about equal size based on their average foil rank , 
and this division was cross-validated*. Table 19 gives the first two of 
these comparisons. The code C* is used to refer to Group B’s clusters 
(see: p c 9^)0 

The rule used in Table 19 (see: p 0 9^)? once again, was the 

most frequent repetition of items within clusters for Group A and Group 
Bo Only four of the 12 items (33 per cent) which were grouped by the 
advance classification and retained this grouping; in the clusters were 
found to cross-validate in Group B 0 Thus, the advance classification 
holds up about as well (or as badly) in cross-validation as it did in 
the clustering. 

Comparing cluster by cluster, there were ten items (33 per cent) 
in the clusters from Group A which occurred in equivalent clusters for 
Group Bo Cluster C* contained two members from C. but also had all of 

5 5 

in it (items 15 ? and 18) which leaves the definition of CJ. ambiguous © 
In any case, it was evident that the clusters themselves did not 
cross-validate any better than the advance classi fi cation G 


TABLE 19 
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CROSS-VALIDATION OF THE RIGHT ANSWERS OF 
GROUP A BY GROUP B FROM THE ADVANCE 
CLASSIFICATION AND THE ITEM CLUSTER 




G roup 

A 

I 

! 

! 

i 



Group 

B 


Cluster 

t 

\ 

\ 

\ 

Item 

i 

i 

s 

L 

Cluster 

1 

f 

J 

. ! 


Item 


c i 

I 1 *a 

t 

2* 

8 

28; 

t 

c 1 

1 

t 

: I* 

i 

2* 

12 


c„ 

i 3* b 

! 

17* 

30 

! 

! 

! 

! 

C 2 

i 3 * 

! 

14 

17* 

29 

c„ 

A 

| 4* 

! 

13 

6* 

! 

? 

1 

1 


| 4* 

? 

26 

6* 


°4 

i 5 

! 

14 

19 

! 

! 

! 

! 

C 4 

i 

; 5 

i 

10 

13 


C 5 

! 7* 

i 

22* 

23 

1 

! 

! 

! 

? 

C 5 

J 7* 

i 

15 

18 

22* 

C 6 

1 9* 

! 

27* 


1 

t 

1 

I 

c 6 

j 8 

i 

27* 

9* 

30 25 

Co 

1 

{10 

! 

11 

16 

29J 

! 

C ) 

? 

! 1] - 

1 

19 



CO 

o 

! 

JI2 

i 

20 

26 

! 

? 

t 

! 

C B 

? 

{ 16 

! 

28 



Cq 

y 

? 

{15 

? 

18 


! 

1 

! 

C 9 

{ 20 

i 

21 



c io 

J 21 
? 

_i 

25 

24 

? 

f 

1 

1 

! 

A 

C 4 

| 23 

! 

! 

t 

24 




a 0 The numbers in I talics cross-validate the advance classification 0 
bo The starred (*) numbers cross-validate the item clusters 0 


Table 20 (see: p. 95) has the right answer clusters arranged by 
their rank based on the average total -correct sco?’e as given in Table 1 6 
(see : p 0 88). and arranged into groups of three clusters with C^ (items 
] 5 and 18) dropped© The cross-validation was then repeated© 

Instead of 33 pen cent there is now 57 per cent cross-validation 
although with only three groups to match instead of nine© An increase 


TABLE 20 


CROSS-VALIDATION OF ITEMS, GROUPED BY AVERAGE FOIL RANK 





Group A 

Proportion 




Group B 






Cross-validating 






Cluster 





Cluster 




High Group 

C 5 

z a 

22 

23 


C 5 

7 

18 

15 22 


C 1 

1 

2 

28 


c' 

1 

1 

2 

12 


C 6 

9 

2? 


High: „78 

C 6 

8 

ZZ 

9 30 25 

Kiddle Group 

C 8 

12 

20 

26 







C 10 

21 

25 
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Low Group 
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16 29 
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13 
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4 

13 
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TOTAL: 0 57 









These 

clusters did not cross-validate: 
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2 3 Ik 17 29 
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C 10 24 






a 0 Items in Italics cross -validated, by group, 
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of this sort would be expectedo More interesting is the distribution 
of the shifts between Group A and Group B clusters,, If the pattern 
for Group A is taken as a reference and the order is considered to 
be fixed, a shift is considered to be + 1 if the mobile item in the 
Group A cluster is found in a Group B cluster associated with items 
one cluster higher for Group A„ Thus, a + 1 shift occurred for item 13 

i 

in clustered with item 5 in where the latter is in 0^. Similarly, 
item 10 in would represent a - 1 shift 0 Table 22 summarizes these 
shiftso 


TABLE 22 

MOBILITY OF ITEMS BETWEEN GROUP A 
AND GROUP B IN TERMS OF SHIFTS 


0 Shift 
Items 

t l Shift 
Items 

- 2 Shifts 
Items 

- 3 Shifts 
Items 

Larger 

Items 

Shifts 

Shift 

1 

7 

3 19 

12 

1 — 1 

ON | 

23 

-4 

2 

9 

8 20 

14 

28 

24 

+4 

4 

11 

10 21 

25 

30 

26 

-5 

5 

22 

13 29 





6 

27 

17 






10 


9 


3 


3 


3 


p 
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Table 22 (st 


9 6 ) shows that the mean absolute shifting of 


items between the two groups was slightly more then one place in the 
hierarchy (s - l<Jl)o This shifting was about one rhird of the shift 
expected if the items had randomly rearranged from Group A to Group B 0 
If all possible shifts were equally probable, the mean shift would be 
3.24. In fact 22 of the 28 items shifted two steps or less, which 
accounts for 79 per cent of the i terns 0 

A possible explanation for these shifts can be found in the 
hypothesized multiple interpretations. This hypothesis would suggest 
that in spite of the homogeneity of these two groups based upon total- 
correct scores, these two groups were obviously not homogeneous when it 
came to clustering of the items into item-homogeneous subtests 0 The 
clustering was based upon correlations which were sensitive to marginal 


totals o The stability of these marginal totals could be expected to be 
affected by the range of interpretations of the items within the groups 
concemedo This shifting could, therefore, be a product of the 
heterogeneity of the examinees and the effectiveness of the item in 
communicating a limited range of possible interpretations 0 


Cross - va l i da t i on of wrong answers 0 An identical procedure to 
the one used for right answers was* used with the wrong answers D 
Table 22 (see: p. 98) gives the cross-validation between the advance 

foil classification and the individual foil clusters between Group A 
and Group B. 

In Table 22 we find that of the advance classification only 
three foils out of 1 6 (or 19 per cent) cross-validate as compared with 
the 1 6 out of 60 foils (or 2 6 per cent) which help to identify the 
clusterso Once again, the advance classification and the clusters 


CROSS-VALIDATION OF ADVANCE FOIL CLASSIFICATION AND FOIL CLUSTERS 
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cross-validate in about the 


same proportion * 


Wrong answers seem to 

cross-validate about as well as right answers. 

The third comparison, once again, was between foil clusters 
grouped into three groups based upon the average foil rank® Table 23 
(see: p. 100) gives this comparison® 

In Table 22 the high group again cross-validated best using the 
hierarchy® With being dropped as uninterpre table, 2? of the remain- 
ing 51 foils (or 53 per cent) cross-validated compared with 1 6 out of 
28 (or 57 per cent) for right answers, and for a combined total of 43 
out of 79 alternatives (or 5^ per cent)® 

Although the wrong’ answers showed a wider range of shifts, 30 
foils had a shift range of - 3 or less (or 59 per cent) compared with a 
probable s = 4®83 for random shifts. Foils, though less stable, showed 
the same trend toward stability as right answers® Cnee again the mult- 
iple interpretation hypothesis could account for the lack of stability. 

The joint cross-validation combining right and wrong answers by 
group were: high group 21 out of 28 ( or 75 per cent); middle group 14 

out of 28 (or 50 per cent); and low group 8 out of 23 (or 35 per cent)® 
That is, the stability increases by about the same proportion (50 per 
cent) from low to high® This increase in stability was also found 
among the wrong answers for the Proverbs Test (Cf Powell, 1968 )® 


The Prediction Value of the Experimental Test 

In addition to the scores and individual responses on the 
experimental test, two achievement scores were obtained for most 
examinees. There was some loss of data, making Group A have 125 members, 
and Group B have 120 members® 

The first achievement test (Achievement Test i) was given with 


TABLE 23 


CROSS-VALIDATION BY GROUPING OP WRONG ANSWER CLUSTERS 


Group A 


Group B 


Cluster- 


Proportion 

Cross-validating 


Cluster 


W 1 

1D i 

—1 

2£D 

w - 
13 

21D_| 

28D 2 

23 Dp 

w, 

o 

9Di 

21D., 

21 Dq 
■ j 

W 5 

12D q 

11D 2 

12D 2 

w 

w o 

7 

- D 3 

28D 1 

iod 2 

W 14 


22 D, 

9d 2 

W 15 

— D 2 

24D 3 


W 3 

4D 

26D n 

* D 2 

W 8 

111 

28D 3 

14D 2 

W, 
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30D 3 



W 

10 

29D-, 
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11 

18D 1 

?d 2 

6d 3 

V 

*12 

19 D X 

24D 2 



High 

Foils 


Middle 

Foils 


now 

Foil:: 


8D n 1?D 1 


?D 3 26D 3 


High 
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2D 2 10D 3 


Low 
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W 15 
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The following clusters did net 
cross-validate 


Unclassified Foils: 
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1 
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1D. l 22D 3 10D 3 18D 1 
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9D„ 24D 12D„ 

d d J 
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W^: 10D 1 , 19D 3 , HD 3 , 13D 3 , 12^, 15V y 181) 2> 26D 2 , 8D 3 
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the experimental test as a subtest for a midterm examination*. The 
scores used for predictive purposes do not contain the experimental 
test scores* The second achievement test (Achievement Test II ) was the 
final examination in the same course. The relationship among these 
tests is presented in Table 24. 


TABLE 24 

CORRELATIONS BETWEEN THE TESTS IN THIS STUDY 



Experimental 

Achievement 

Achievement 

Test 

Test I 

Test II 


Experimental 

Test 

1.000 



Achievement 
Test I 

.224 

1.000 


Achievement 
Test II 

o!25 

( — 1 

1.000 


As shown in Table 24 the two achievement tests were moderately 
correlated (r = .414). The relationship between the experimental and 
achievement tests was considerably less, particularly with respect to 
Achievement Test II*. 

In order to establish the predictive validity of the 
subdivisions of the experimental test, several comparisons were run 
using a step-wise multiple regression technique in all cases. Each 
achievement test was predicted separately for each of Group A and 


Group B. The following predictions wore made. 
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1. Total-correct score on experimental test predicting the 
total- correct scores of the achievement tests. 

2. Right-answer subtest scores on the experimental te’st with 
the subtests defined by the advance classification of items 
predicting the total-correct scores on the achievement tests. 

3. Combined right answer subtest scores and wrong answer 
subtest scores with the subtests defined by the advance 
classification, predicting the total-correct scores in the 
achievement tests . 

4. Scores on right-answer subtests defined by interpreted 
clusters used to predict the total-correct scores on the 
achievement tests . 

5- Scores on combined right-answer* and wrong-answer subtests 
defined by all interpreted clusters used to predict the 
total- correct scores on the achievement tests. 

6. Scores on right -answer subtests defined by grouping right- 
answer clusters by the foil hierarchy used to predict the 
total- correct scores on the achievement tests. 

7* Combined right- and wrong-answer subtest scores defined by 
grouping of clusters on the basis of the foil hierarchy 
used to predict the total-correct scores on the achievement 
te s t s o 

In each case the total- correct score of each achievement test 
was being predicted® 

Table 23 (see: p. 103) gives the results of these predictions 


for Group A 


TABLE 25 


] 03 


STEPWISE REGRESSION OP GROUP A DATA USING 
SEVERAL COMBINATIONS OP VARIABLES 


! 

! 


Combinations 

; of Variables 

,a 


1 

1 

I 

\ 

2 

3 

4 

5 

6 

7 

2 ' 

Achievement Test I R [ 

.059 

.142 

.228 

.068 

.278 


.055 

R ! 

. 1 

.243 

.377 

.478 

.261 

.525 

— 

.234 

1 

No, of Predictors available} 

1 

5 

14 

6 

18 

3 

6 

No. of Predictors used"* 3 

. _ 1 

1 

5 

14 

3 

18 

None 

2 

2 1 

Achievement Test ll R j 

.006 

.029 

.130 


.156 

. 04-9 

.085 

R ! 
1 

<r\ 

0 - 

0 

0 

.169 

.360 

— 

.395 

.222 

.291 

! 

No. of Predictors available} 

1 

5 

14 

6 

18 

3 

6 

1 

No, of Predictors used J 

4 

1 

l 

9 

N one^ 

11 

0 

5 

1 

Total number of right } 

answers used [ 

t 

on 

30 

30 

1 Q 

1 Q 

28 

28 

! 

Total number of foils used J 

! 

L 

0 

0 

7° 

0 

46 

0 

46 


a. The combinations of variables are defined by number as given on 
page 102. 

b. Only those predictors which made a significant contribution (£ [_ .06) 
to the prediction were included on this table. 


Table 25 gives the correlation between the total-correct score 
on the experimental test, the multiple correlation achieved for signif- 
icant variables (jd j_ .06). The squared multiple correlation (R ) are 
also given to indicate the amount of variance accounted for in the 
predictions. 

A similar set of data for Group B follows in Table 2 6 
(see: p. 104). 
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TABLE 2 6 


STEPWI SE REGRESSION 
SEVERAL COMBI NA r J 

OF GROUP B DATA USING 
.’IONS OF VARIABLES 



! 

! 

t 

1 

t 

i 

Combinations of Variables a 
2 3 4 5 6 

7 

Achievement Test I R" ! 

i 

i 

R 

i 

.046 
. 214 

0 063 
.251 

.190 

.436 

.035 

.188 

.189 

.435 

— 

.091 

.301 

No. of Predictors available' 

1 

5 

14 

6 

18 

3 

6 

No. of Predictors used ' 3 

t 

1 

3 

14 

1 

14 

None 

5 

Achievement Test II R‘" j 

.046 

.082 

.135 

.054 

.237 

.057 

.193 

R ! 

t 

.214 

.287 

.368 

.232 

0 487 

.239 

.439 

i 

No. of Predictors available j 

1 

5 

14 

6 

18 

3 

6 

1 

NOo of Predictors used J 

. . . . , ! 

1 

4 

9 

2 

18 

O 

/ 

O 

1 

Total number of right J 








answers used | 

i 

30 

30 

30 

19 

19 

28 

28 

t 

Total number of foils used j 

0 

0 

70 

0 

46 

0 

46 

a. The variables used with Group B 

were identical in 

definition 

to 


those used in Group A. 

b. Only these predictors which made a significant contribution 
(p. L °C 6 ) to the prediction were included in this table. 

There are two considerations relevant to Tables 25 and 26. 

First, the value of the procedure is partly determined by the amount of 

r\ 

variance accounted for (as given by R ). Using this criterion, there is 
a consistent improvement in prediction when the scores on wrong-answer 
subtests are included in the analysis. When this v/as done, the wrong- 
answer variables, in general, accounted for more of the variance than 
the right-answer variables within the same solution. The interpreted 


clusters give a better solution than the advance classification. Both 
of these were better than the grouping by the foil hierarchy. The 
poorest predictor was the total-correct score on the experimental test. 

The second consideration is the statistical significance of the 

2 

improvement of these values of R when compared with each other. The 

formula for this comparison is: 

2 2 
■p p 

F = I ~ ;> X N - V - 1 

1.00 - Rj 2 K - L 

■where N is the number of persons, 

K is the number of independent predictions in R^, 

and L is the number of independent predictions in R^. 

This procedure gives the usual M F M test with degrees of freedom, 

N - K - 1 and K - L respectively. The results of these calculations 
derived from the data in Tables 25 and 26 (see: pp. 103 and 104) are 

given in Tables 27 to 30 which follow (see: pp. 106 to 109). 

Table 5 1 gives the significance of the difference between the 
predictions of Test I for Group A. It shows that the sequence 1 / 2 = 
3/4/5 stands clearly in the diagonal. One factor which may be 
involved in the equality 2=3 may be the fact that the number of 
predictors increases from 5 to 14. If all other values remain the same, 
an increase in the size of the sample of less than 50 per cent would 
make the difference between combination 2 and 3 significant® Also, no 
variables made a significant contribution in combination 6, and 2 
variables were significant in combine.! ion 7? which makes 7 a signifi- 
cantly better predictor than 6. Thus, for Group A when predicting Test 
I there is a consistent tendency for the combined right and wrong 
answers to be better predictors than the right answers alone. 
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TABLE 27 

SIGNIFICANCE OF DIFFERENCES BETWEEN R 2, S 




FOR GROUP A 

WIEN 

PREDICTIN' 

G TEST I 



E l 2 

! 

1 

i 

V 

; 

f or 

Variable 

Combinations 3, 


for variable 
comb inations 

i 

l 

; i 

2 


3 

4 6 

7 


! Z 

t 

2 b : z 

♦ 

r! ; 

t 

l r! . 

f 

p r! z 

1 _ 

r! Z r 

1 
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i 
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1 

1 

i 

t 

i 

t 

? 

i 

i 

i 

i 

i 

1 

1 

! 

! 

! 

[empty 
! cell 

— J-ve 

t 

2 

t 

| 2. 90 

J 

\ 

.05! 

i 

! 

1 

t 

1 

f 

t 

1 

!5 

i 

.41 .0l| em R y 
! cell 

! 

— {4.02 .01 

I 

3 

l 

j 1 . 86 

„i 

! 

.05! 1.36 

I 

1 

1 

1 

“ ? 
. I 

1 

l 

f 

! 

t 

! empty 
! cell 

! 

— J2.05 .05 

4 

i 

| .53 

t 

1 

t 

1 

1 

1 

J 2 

i 

1 

.0? . 05 ! 
! 

t 

{empty 
! cell 

! 

— J 1 .69 — 
1 

5 

! 

t 

! 

t 

? 

1 

1 

1 

1 

i 

i 

i 

i 

i 

! 

! 

! 

! 

f 

1 

| empty 
! cell 

! 

— '[2.05 .05 
! 

7 

1 

1 

1 

1 

t 

! 

! 

1 

I 

i 

i 

\ 

! 

! 

! 

1 

! 

! 

! 

J 

1 

J empty 
! cell 

1 

t 

! 

1 

1 . 



a. Definitions for these variable combinations are given on p. 102. 

b. Only the probability ('£ level) for significant differences are shown 0 


Table 28 gives the significance of the differences between the 
predictions of Test II for Group A (see: p. 10?). 

There is no similar pattern when predictions are made to the 
future Test II as compared with the concurrent Test I. Some of the F 
values in the equivalent diagonal are large enough that a larger sample 
might make them significant. At least all predictor combinations are 
better than the total-correct score. Although the grouping of 
alternatives (combinations 6 and 7) c-n the basis of the hierarchy does 


107 


TABLE 28 

SIGNIFICANCE OF DIFFERENCES BETWEEN R 2, S 




FOR GROUP A WHEN 

PREDICTING TEEj 

p ii 



E l 2 

l 

l 

! 

! 






for variable 

1 

! 

R for 

Variable Combi 

.nations 3 " 



combinations 

1 

1 

2 






! 1 
i 

2 

3 4 

6 

7 



' 

: f 

i 

FI F Z\ 

1 1 

F £l F 

f 

El Z 

! 

e! f 

! 

El 

1 

i 

i 

! 

1 

t 

f 

! 1 

t t 

1 1 

t ! 

! 1 

1 

1 

1 

1 

1 

1 

t 

! 

f 

i 

1 

1 

! 

! 

t 

1 

1 

I 

! 

1 

2 

1 

1 *c 

1 

1 

! 

1 1 

1 1 

1 1 

1 1 

t 1 

1 

1 empty 
1 cell 

t 

1 

f 

t 

1 

1 

1 

1 

1 

1 

I 

1 

! 

3 

1 

{2.05 

j 

1 1 

.0511.72 — j 

1 1 

1 

1 empty 
! cell 

1 

— *1.53 
\ 

t 

-11.49 

| 

j 

1 

j 

! 

1 

5 

! 

1 

12.01 

! 

.0511.74 — ll 

! ! 

i 

J empty 

* 6 ' 1 cell 

1 

-- 1 1 . 59 
! 

1 

— 11.57 

f 

\ 

1 

1 

6 

1 

! 

1 C C O 
! J • .)' 

f 

1 1 

1 t 

AdO ^ ! 

. J .. , • • ^ i j 

1 . ! 

i 

[empty 
1 cell 

t 

! 

! 

I 

t 

1 

1 

1 

! 

J 

1 

1 

1 

( 

1 

7 

i 

j2.57 
_] 

1 1 

.05 J 1.82 — } 

_i i 

1 

[empty 
! cell 

1 

— |1.56 

L 

! 

1 

f 

" ! 

l 

! 

! 

1 

! 


a. Definitions for these variable combinations are given on page 102. 

b. Only the probability (n level) for significant differences are shown. 

c. * means: Denominator 0, F value indeterminate . 


not yield significant differences, these two variables tend to have the 
largest F values. Once again, a larger sample size might have made the 
difference 0 The cross-validation of the multiple regression coefficients 
which follows this section casts further light on this aspect of the 
problem o 

Table 29. (see: p. 108) gives the significance of the differences 

between the predictions of Test I for Group B. 
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a. Definitions for these variable combinations are given on page 102. 

b. Only the probability (jj level) for significant differences are shown. 

c. * means: Denominator 0, value indeterminate 0 


Table 29 is very similar to Table 28 except that none of the 
predictor combinations are signif icantly better than any other including 
the total-correct scores,. These findings are not surprising considering 
the low level of cross-validation already found for all combinations 
except the grouping based upon the hierarchy. Although none of the 


values for these groupings (Combinations 6 and 7) arc significant, these 
two combinations have the highest F values. Once again a larger sample 
size might have made the difference 0 

Table 30 gives the significance of differences between 
predictions of Test ]1 for Group 13. 
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a 0 Definitions for these variable combination^ aie gi'/en on P-*ge 1C.-® 
b. Only the probability (p> level) for significant differences are shown. 


c. * means: Denominator 0, E value indeterminate. 


Table 
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30 shows Combination 7 to be a highly significantly better 

predictor in four of the six cases. When comparing Combination 3 with 7? 
2 

the I? is larger for Combination 7; but the number of significant- 

predictors is larger for Combination 3 and, for this reason, gives a 

negative F value in the formula. Logically, Combination 7 would, 

therefore, be the better predictor in this case as well. Combination 5 

2 

has a somewhat larger R but requires many more predictor variables to 
achieve this, making the difference clearly insignificant. 

Three trends seem to emerge from this data. First, combined 
right and wrong answers generally seem to yield the best predictions 0 
Second, the best prediction of a concurrent test seems to be 
found by using the interpreted clusters for the same group « 

Third, when predicting remote events or the results of another 
group combining the predictor variables on the basis of the hierarchy 
would seem to give the best predictions. 

Cross -validati on of the multiple regress j on coefficients . It is 
possible to cross- validate multiple correlations by finding the vector 
product of the validity coefficients for one group and the standardized 
regression weights for the same variables from the other group as 
follows: = V ! W 

where V’ is a row vector of the validity coefficients for 
one group, 

W is- a column vector of the standardized regression for the 
corresponding variables from the other group, 

2 

and R is the resulting vector product. 
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Since the combining of right, and wrong answers seemed to give 
the best results, only these combinations of variables were cross- 
validated in this manner. Table 31 (see: p. 112) gives the results of 

these calculations . 

To begin with, Table 31 shows that Combination 3? the advance 
classification, does not survive cross-validation. Clearly, the best 
predictor of the concurrent test for Group A was the interpreted 
clusters (Combination 3)> a finding consistent with the findings for the 
significance of differences. This combination (5) of variables did not 
cross-validate in any other situations. 

For the remote test, Combination 7 proved to cross-validate very 
well, much better than the significance of the differences would suggest. 
For Group B, Combination 7 cross-validated about as well as the propor- 
tion of item-for-item cross-validation would suggest that it should. 

These findings also tend to support the suggestion that grouping on the 
basis of the hierarchy may be the best method for predicting future 
performance or performance in another group. 

The reader is cautioned that the correlations between the 
total-correct scores of the experimental test and the achievement tests 
suggest that. these tests may be dissimilar in the characteristics they 
are measuring. This situation would be expected to produce lower 
multiple correlations than tests of greater similarity might achieve. 
Second, near zero multiple correlations arc easier to cross-validate 
than higher ones, so that the relative stability of these correlations 
can only supply suggestive results by themselves 0 
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CROSS-VALIDATION OF MULTIPLE REGRESSION COEFFICIENTS 
FOR COMBINED ANSWERS 
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Summary of Chapter \ r 


Briefly, the findings as reported in this chapter were as 

follows : 

lo Interpoint distance gave the best statistical solution to 
the data being considered in this study. 

2 0 Logi co- semantic analysis provided reasonable support to the 
construct validity of the procedure used, provided that 
alternative classifications are permissible, and inter- 
pretations were confined to the group under study. 

3. There may be a hierarchy of foils which parallels Bloom's 
Taxonomy which may influence the way in which items perform. 

4. None of the- cross-validations were very strong, with the 
hierarchy tending- to be somewhat better supported than other- 
aspects of the analysis. 

o Wrong answers, in general > tended to add significantly to 
the predictions of both concurrent and future achievement 
whenever significance was found. 

60 Within one group the interpreted clusters gave the best 

concurrent prediction, otherwise the grouping ori the basis 
of hierarchy seemed to produce the best prediction D 


CHAPTER VI 


CONCLUSIONS AND IMPLICATIONS 

The conclusions which can be drawn from this study are discussed 
in three sections*, first, the conclusions which are relevant to the 
experimental test used in this study, second, the conclusions which are 
relevant to the analytical procedure, and finally, those v/hich are 
relevant to the syst emati c- res reuse n^styl a te c 

The implications which can be drawn from this study are discussed 
in four sections e First, there are the implications of the results of 
the use of this analytic procedure to the theory of test analysis 
procedures c Second, there are the implications to the design, construc- 
tion, and interpretation of taxonomic tests „ Third, there are a number 
of implications of this study to educational practice 0 Finally, this 
study has opened enough possibilities for future research that these 
are discussed in a separate section 0 

Conclusion s Related to the Experimental Test 

Superficially, the experimental test used in this study would 
seem to have been a weak instrument but, as will be seen, the criteria 
usually used for evaluation may not have been applicable to this teste 
Using the usual criteria, for instance, the selection ratio for the 
right answers on most of the items was low; the biserial correlations 
of the items to the total correct scores were also low relative to the 
size of the corresponding difficulty ratios; the internal consistency 
based upon the Kuder-Richardson procedure was low, and the interrater 
reliabilities left a great deal to be desired 0 

On the other hand, the usual criteria employed for the 


evaluating of tests may not be appropriate for this one 0 Briefly, the 
desirability of middle difficulty items is a criterion based upon the 
assumption that this level of difficulty maximizes the discrimination 
of the test when all the items are dichotomous variables G When all 
alternatives are being considered rather than when the item is being 
considered either right or wrong, this criterion seems no longer to 
apply o 

The biserial correlation of the test items taken against the 
total correct scores is a criterion of the di s criminating power of 
these items assuming that the test as a whole is highly homogeneous 0 
The low internal consistency of this test suggests that the test was 
not homegeneouSo Evidence for the lack of homogeneity in this test can 
be found in the logic of the construct model (Bloom’s T axonomy ) , and in 
the fact that the test subdivides in the cluster analysis into ten 
clusters of right answers, at least six of which were nearly orthogonal. 
Also, the loadings of the items on these nearly orthogonal factors were 
very nearly the original lengths of the vectors in the principal axis 
matrix from which they were derivedo This latter result suggests that 
the internal consistency within clusters was substantially higher than 
within the test as a whole 0 For these reasons, the usual criteria may 
not apply to this test 0 

On the positive side, the average total correct score for 
persons selecting each alternative was higher for the right alternative 
than for all others used by at least 1„5 points in 37 out of 58 cases 
(or 64 per cent) which gives some support to the strength of the test 
(see : Table 40, p. IfO)* Although the clusters did not cross-validate 

very well, when the shifting of items into other clusters was considered, 
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it became evident that the items and foils were much more stable than 
would be suggested by chance alone 0 Also, the clear evidence for an 
interacting hierarchy of items and foils would seem to provide strong 
support for the possibility that the instrument was measuring some 
systematic characteristics of the examinees 0 

Precisely what these characteristics were would seem to be more 
ambiguous than the probability that they were being measured. They were, 
however, clearly process-oriented characteristics 0 The doubt about 
precise definitions arose from two sources, l) the lower than desirable 
interrater reliabilities, and 2) the lower than desirable cross- 
validation of the clusters. Both these weaknesses in the interpretability 
of the results, however, may be a property of this type of test, and not 
a criticism of it Q 

If the proportion of the total variance used in the factor 
solution is considered, there was 37°? per cent accounted for by the six 
factors for the items and another A8.6 per cent for the 15 factors used 
for the wrong answers, suggesting a higher internal consistency than the 
Kude r-Ri chard s on results s ugge s ted 0 

Thus, the instrument displays the following properties: 

I® Significant] y improved prediction of independent concurrent 
achievement scores for the same group of examinees using 
interpreted clusters from both right and wrong answer clus- 
ters combined over the other combinations of scores testedo 
2 0 A clear hierarchical pattern of both items and foils 0 The 
right answers and the foil selections seem to interact and 
to be relatively stable within the hierarchy under cross- 
validation when the range of shifts are consideredo 


3o An overall discrimination apparently based upon process - 
oriented events rather than content-oriented events 0 
The construct objective was to produce a process-oriented 
taxonomic test which had predictive value relevant to achievement 
variables. Whatever criticisms might be made of the test, the results 
made it clear that it met this construct objective to a reasonable 
degree, and for this reason be taken to be a valid test c Also, the . 
indirect evidence suggested that the test was probably more reliable 
than was suggested by the direct evidence 0 Precisely which procedures 
should be used to establish validity and reliability estimates for tests 
of this type are not yet clear. 

Conclusions Related to the A nalyti c Procedures Used 

The analysis began with phi coefficients 0 The use of these 
coefficients can be defended on the grounds that none of the assumptions 
which are made for these coefficients was violated by their use© The 
two variables being related for each coefficient are discrete since each 
represents the selections made by the members of the same group for dif- 
ferent alternatives o They were dichotomous because an accept-re ject 
decision applies to all alternatives as a requirement of the response 
procedure. Linear dependencies were removed from the data by parti- 
tioning the matrix o Finally, since all values were expressed as 
frequencies of occurrence, the categories were amenable to appropriate 
representation by two-point values 0 

The resulting large matrices of phi coefficients were simplified 
by principal axis factor analysis in order to remove as much measurement 
error as possible, and to maximize the variance accounted for by any 
particular number of dimensions in the space being used© Beyond this 
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point the procedures seemed to separate into two aspects, those which 
are commonly -used for the study of tests at id their results, and those 
which are not commonly used hut which are specifically selected to meet 
problems which may arise in the interpretation of the results. 

The result of this study derived from the commonly employed 
procedures was uniformly ambiguous, inconclusive, or negative, whereas 
the less common procedures uniformly produced significant results 0 As 
indicated, the test Itself would seem to have been of questionable 
value if it were evaluated by the commonly used procedures, and yet it 
clearly met the construct properties it was designed to meet 0 

The Procrustes rotations of the factor matrix to fit either 
content or process in the advance classification produced negative 
results o The results of the usual analytic rotations on the factor 
matrices were not reported in Chapter V because they were as unintcr- 
pretable as the Procrustes rotations 0 However, when interpoint distance 
clusters were used to avoid the problem of rotation, a set of nearly 
orthogonal groupings of the variables was produced 0 Unquestionably, the 
cluster approach produced a statistically satisfactory representation of 
the data leaving the researcher with the problem of interpretation to be 
resolved by non-statistical methods. 

Cross-validation of clusters was equally disappointing, and 
contradicted the results of the t-test for uncorrelated samples based 
upon total correct scores 0 Substantial improvements in the proportion 
of cross-validation were found when the apparent hierarchy of items and 
foils was taken into account 0 Additional support for the cross- 
validation was found in the pattern of shifts of alternatives which 
occurred among the clusters between the two groups 0 The data were much 
more systematic among the clusters between the two groups 0 The data 
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more systematic than the usual procedures seemed zc suggest 0 

Using total -correct scores to establish a hierarchy of foils on 
both a within item and a between item basis produced results with a 
moderate relationship® Mien these two procedures were used to order 
the right answer clusters, however, a highly significant similarity 
between these latter two orderings was found, leaving little doubt that 
an interactive ordering between the right and wrong answers was a 
systematic characteristic of the datac This ordering was unrelated to 
the total correct averages for the right answer clusters and to item 
difficulty, making the use of total-correct scores for the establishment 
of the hierarchy questionable® However, the shift patterns of the cross- 
validation suggested a much more stable result than did the cross- 
validation alone, suggesting that these shift patterns might be used to 
determine the ordering of the clusters instead of using total-correct 
scores . 

Although all predictions were low, the use of the interpreted 
clusters did tend to give significantly better concurrent prediction of 
the total -correct scores on the independent concurrent achievement 
measure for the group on whom the interpretation was attempted® The 
amount of variance accounted for increased roughly three times in this 
case & The question of the validity of the use of the total-correct 
scores as adequate representations of achievement on the achievement 
measures was not explored. Finally, the hierarchy also proved to he 
more broadly stable during cross-validation than other variables 0 

Evidently, the more conclusive results were found by the less 
common procedures ® Since these procedures were designed to fit the 
specific problems raised in this study, and since the statistical 
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adequacy of these procedures proved to be beyond question, they would 
seem to be, collectively, a more adequate method of approaching the kind 
of data this study produced than the more common procedures would seem 
to be G 

Conclusions Related to the Systematl c Response Hypothesis 

The adequacy of the experimental test, and the analytic 
procedures uouu. j.cr one purpose this sludj seem to be establisned to 
a reasonably acceptable level. Taken alone, the data were sufficiently 
systematic in both right and wrong answer matrices bo establish that 
M most if not all of the answers given to multiple choice achievement 
tests are selected upon a systematic basis 11 may be a reasonable approach 
to human performance 0 The tv/o findings most relevant were the presence 
of the interactive hierarchy and the increase in predictive validity 
evident when wrong answer clusters were included in the regression 
equations o 

If the evidence supporting the multiple interpretation hypothesis 
is included, the support for this psychological postulate is greatly 
increasedo To begin with the negative results from the Procrustes 
rotations for the advance classification by both content and process 
established the inadequacy of this approach 0 Logico- semantic analysis 
clearly established that process variables provided the best interpreta- 
tion of the clusters. But the failure of the Procrustes rotation with 
the advance classification, the low cross-validation levels and the 
shift patterns clearly indicated that the classifications are not 
mutually exclusive 0 

Kropp et al (1966) found that the same subtests were differently 


defined by the hit, for different grade levels, and Powell (1968) 
reported only about 6 0 per cent cross-validation by reported reasons 
for the selection of particular wrong answers* Also, the higher level 
alternatives tend to be more stable than the lower ones when taken in 
combination, which is consistent with other findings (see: Powell, 1968), 

Thus, independent studies also report findiiigs suggesting that the 
classifications of answers may not be mutually exclusive 0 A reasonable 
synthesis of these findings would be to suggest that each item may be 
interpreted in a variety of different ways D That is, the poor showing 
on cross-validation and the lower than desirable interrater reliabilities 
may be a product of multiple interpretations of the items 0 Strong 
support for this hypothesis was found in this study in the systematic 
character of many of the shifts which occurred among the alternatives 0 
These shifts wore sufficiently small in range for most items that the 
possibility of their use in the establishment of order among the 
variables could be proposed (see: p*> 9 ?)* Further support was found 

in the fact that the prediction of the concurrent test on the inter- 
preted group was the only case where the interpreted clusters had a 
distinct advantage o Predictions based upon the hierarchy, on the other 
hand, seemed to be less powerful, but more stable over a broader range 
of time and population c These findings support the multiple interpreta- 
tion hypothesis as well because they suggest the short range 
applicability of specific interpretations 0 

Perhaps the range of this applicability of interpreted findings 
could be increased if the heterogeneity of the examinees upon whom the 
interpretations are made were reduced 0 

In summary, the conclusions of this study were: 
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1, Human per To nuance , when abstracted from responses to 

multip'Je choice achievement tests involving higher mental 
processes, would seem to he systematic, and to display 
evidence of multiple interpretation of the communication© 

2 0 There would seem to be a hierarchy of foils which parallels 
the hierarchy of right answers and which influences the way 
in which each total item performs © The levels of the foils 
themselves seem to depend upon the systematic ways in which 
this totality of each item is approached. 

3. Wrong answers contain potentially useful information with 
respect to achievement when higher mental processes are 
involved. 

Before the implications of this study are discussed, a statement 
should be made concerning the limitations to generalizability apparent 
in this study. 

Limitations to Gene ral i z ab i 1 i t v 

There are several restrictions to the generalizability of the 
findings of this study which can be derived from the nature of the study 
and its conclusions 0 First, the findings of this study do not apply to 
multiple choice achievement tests where the simple recall of informa- 
tion is the only characteristic being measured© The experimental test 
was process rather than content oriented and Knowledge level items 
were considered inappropriate to its format, hence this limitation 0 

Second, the findings of this study do not apply where the cost 
of the additional effort required to obtain and interpret categorical 
information upon the examinees is greater than the cost of information 
loss, and possible mi sclassi f ication attendant thereto, by using the 
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much simpler total-correct score method of evaluation o 

Third., these findings may not apply when the most competent are 
being screened from already competent individuals for some specific 
purpose o A stronger statement in this respect cannot be made because 
later research may show that wrong answers may supply valid information 
for the purpose in question. For instance, Irr (irrelevancy) foils may 
identify the most creative individuals among the high performers 0 

Finally, if a researcher has a valid reason to evaluate the 
effectiveness of a single treatment given to a heterogeneous group by 
using a single ordinal dimension for the particular group in question, 
the findings of this study clearly do not apply Q 


Implications of This Study to the Theory of Test Analysis 

There are several situations where the findings of this study 
are very important to test theory 0 The fact that the more common 
procedures tended to give ambiguous, inconclusive or negative results 
raises a number of pertinent questions. 

Classical test theory begins with the assumption that 
X. = T. + E. 

ill 


th 

where X_. is observed score of the i individual, 
T. is his true score, and 
E_. is the measurement error 

However, for multiple choice achievement tests, this observed 
score (X^) itself is usually a composite entity obtained from the 
summation of single events as follows: 


X. 

i 


n 




x 
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where x. . is binary, being 1 if the i'th 

individual answered the item correctly, other- 
wise 0, and 

n is the number of items in the test* 

The issue then becomes --is sufficiently homogeneous for 

all individuals to justify the use of classical test theory? If it is 
not, as was evident in the present study, then an alternative approach 
to the data would seem to be needed, since the more common approaches 
proved unsatisfactory in this studio 

Within the context of the present study, several considerations 
must be met by this alternative approach© The phi coefficients used 
are extremely sensitive to the magnitudes of marginal proportions * For 
this reason, if particular alternatives are selected for a different 
range o~ reasons among two samples of individuals, it would be expected 
that these alternatives would migrate to new clusters for reasons of 
systematic differences between groups rather than for reasons of 
measurement error 0 Also, if the range of reasons within a group of 
examinees were too broad, the interpretation of clusters would he ex- 
pected to be difficult and possibly not applicable to specific individ- 
ual So That is, the first assumption made would he that different 
reasons for the selection of a particular alternative would be reflected 
by differences among overall patterns 0 These arguments suggest the need 
at the outset for a homogeneous group of examinees© The key to 
homogeneity in this srudy would seem to be the shifts in category which 
occurred upon cross-validation© Perhaps an homogeneous group should be 
defined in terms of minimizing the shifts which occur in the clustering 
of the alternatives for any, or all possible random assignments of the 
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group members to an arbitrary number of groups® In short, the 
procedure should probably begin by selecting groups of individuals with 
maximum cross-validity within these groups® 

The clusters for such groups will be as stable as possible on 
the basis of the determination of their composition® Hence, the 
possibility of interpreting the resulting clusters should be optimized, 
as should the applicability of these interpretations to the individuals 
within the groups 0 

V/ith the clusters thus stabilized it should be possible to 
determine the clusters using all the data rather than a simplification 
of it, since the surface-to-surface interpoint distance (d) between the 
ends of the vectors within the hypersphere can be determined quite 
simply by assuming that phi (jz5) is the cosine of the angle between the 
amis of the isosceles triangle produced by the vector pairs® In this 
case the distance (d) is 

d =v/2(l -"7)~ 

A tighter level of homogeneity is possible among individuals 
who have essentially the same response patterns but these individuals 
would tend to have only one meaningful cluster composed of all cr most 
of their responses thus rendering interpretation impossible,. 

The dimensionality of the data from a homogeneous (by cross- 
validation) group of individuals would be dete mined by the minimum 
number of homogeneous clusters which can be extracted before orthogonal- 
ity between the centriods of the clusters begins to disappear. Thus, 
the proposed procedure as just outlined as related to the problems 
raised by the data in this study provides a unique solution once the 
stop criteria are established® Each cluster would be categorical 


(in the sense given on page 4 ?) ? and optimal i.y in to rpre table assuming 
that differences in interpretation of the communication hy the examinee 
are characterized by differences in selection pattern© 

There may « a.s the evidence from this study suggests, he an order 
among the categories which can he determined by the shifts which occur 
during cross-validation. Poor items would he unstable for the cross- 
validation O 

So far as the categories themselves are concerned, these would 
he expected to he of two types l) nominal, and 2 ) ordinal© Nominal 
categories would he expected to he himodal with the modes tending to 
polarize at the extremities of the potential range within the category© 
Ordinal categories would he expected to display scalability 
characteristics within their potential range© 

Relationships among categories other than the ordinal 
(hierarchical) one could probably he determinable by the relationships 
among the centroids of the categories. For instance, Powell and 
Isbister (1969) found a polarity between Invalid Assumptions for the 
wrong answers and Synthesis items among the right answers© In such a 
case it might he unnecessary to partition the matrix to remove linear 
dependencies . If this latter facilitation could he provided by this 
procedure, a homogeneous test in the classical sense would he one in 
which the right answers foimed a single cluster of the ordinal type. 
Thus, the proposed procedure just given would seem to contain the 
characteristics of classical test theory as a special case© 

It would he reasonable, then, to argue that the findings of 
this study suggest the need for alternative procedures to the ones in 
common use for the analysis of data from multiple choice tests, and the 


findings suggest 


a particular procedure which 


contains the common] y 


used 


procedures as a special case 0 


Implications of This S tudy Concerning Taxonomic Tests 

Bloom (1956) defines his Taxonomy as having three aspects: 

1. It is a classification systern 0 

2c It is hierarchically ordered on a "complexity” dimension,, 

3o Each higher level is formed by combinations of the lower 
levels o 

The two properties of classification and ordering among classes 
combine to distinguish a taxonomy from other classification systems 0 
Thus the evidence from this study supports the description of both 
Bloom 5 s T axonomy and the Guidelines as taxonomies 0 Noting the evident 
interrelatedness of these two taxonomies in this study suggests that 
they may both be part of a single taxonomy. 

Concerning the "complexity" dimension. Bloom said. "Our attempt 
to arrange educational behavior from simple to complex was based upon 
the idea that a particular simple behavior may become integrated with 
other equally simple behaviors to form a more complex behavior /page 18/." 

The findings of this study which produced no relationship between 
total-correct scores arid the hierarchy where right answers were 
concerned, and no relationship between average item, difficulty within 
clusters and the hierarchy did not help to identify the meaning of the 
term "complexity 0 " Since the hierarchy could apparently have been 
produced through cross -validation procedures without recourse to total- 
correct scores, the meaning of "complexity" becomes even more vague 0 
However, the finding that "higher" level members of the taxonomy tend 
to be more stable than lower levels suggests that these categories may 
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be the product of "more powerful strategies 0 11 

The third aspect of Bloom 1 s Taxonomy as noted previously 
(see: p 0 127) also deserves attention 0 This aspect of his definition 

would seem to arise as an hypothesis from his definition, the 
'‘complexity’ 1 dimensio n G About this s ub sump t; j ve property of the 
Taxonomy Bloom himself said that the evidence he could collect to 
support this property was inconclusive (Bloom: p G 19) o 

Other evidence concerning this subsumptive property is meagre 0 
Kropp et al (1966) did not find the clear reproduction of the pattern 
which they expected to find in the factor analysis of their tests 
^/Kropp: p c 91 ff/o Also, their Simplex analysis did not produce the 

consistent order that this property would predict (see: pp Q 24-2.5) o 

Powell and Isbister ( 1969 ) found that a promax rotation did not 
improve the resolution of factors between subtest, scores based upon the 
advance classification for essentially the same test as used in this 
study. The subtests as defined in this study by cluster analysis 
rather than by advance classification showed this same tendency to 
orthogonality 0 It is premature to be dogmatic, but it is possible that 
this subsumptive property may be a hypothesis which will be refuted by 
the evidence Q Alternative analytical procedures such as the one 
outlined above may be needed to settle this issue conclusively 

There are alternative theoretical positions which would predict 
the possibility that strategic categories may be discrete and hierar- 
chically ordered by "power” rather than subsumpt ive 0 Piaget /1963, 

Po 13 rrf, for instance, has suggested that development may involve 
shifts in the schema 17 in which case development may be expected to 
7 

Alternatively, "the acquisition of new strategies,” 

£see: Powell, 19^7? P* 286 ff J* 
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proceed :n a series of discrete phases and stages, each of which would 
be expected to have its own distinctive properties© Such evidence as 
is available, in particular the difficulty in determining a data-based 
definition of "complexi ty H as just discussed; the apparent tendency for 
’’higher" clusters to he more stable than "lower" clusters; and the 
broader cross-validation support for the hierarchy than for specific 
interpretations add suggestive support to the latter alternative over 
the former© 

Thus, the advent, of taxonomic achievement tests raise some 
issues in connection with the analytic procedures and the interpretive 
procedures used for these tests© Whatever else, the results of this 
study have clearly shown that these tests produce a genuine taxonomy 
which might be improved by the systematic development of foils, arid the 
use of the responses to these foils as information when evaluating 
and interpreting these tests and when evaluating, interpreting, and 
predicting the performance of the individuals taking them D 

Implications of Thi s Study to Educational Practice 

The findings of this study suggest that tests which are clearly 
homogeneous regarding internal consistency may form a special case of a 
broader class of tests which have taxonomic properties© This conclusion 
has broad implications with respect to their use in the educational 
settingo 

To begin with, the use of the Guidelines would seem to have 
several practical advantages. First, they simplify item writing because 
they provide a systematic basis for writing a broader range of foils 
than can be made without them© Second, the Guidelines improve the basis 
for the reasons why a foil is wrong c Third, as research further extends 
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the range of Guideline categories and 7*0 fines their definition, it may 
be possible to increase the precision with which such concepts as 
analysis may be defined, further improving the construct validity of 
process-oriented taxonomi c tests. 

Another advantage may arise from the extension of the Guidelines 
into the Misreading and Misclassification types of foil c Such an 
extension may link what is now known about diagnostic characteristics 
of tests in the areas of content-related performance and skill -related 
performance o This linkage may make it possible to extend the diagnos- 
tic aspect of testing beyond the knowledge and comprehension 
characteristics of of reading and arithmetic into the more abstract 
characteristics of mathematics and into the more esoteric subjects such 
as social science, and perhaps oven literary appreciation where the 
subject matter is clearly open to multiple interpretations 0 

If diagnostic testing can be coupled through research with 
improved definitions of educational objectives and the factors involved 
in their attainment, teaching could be more nearly like the practice of 
medicine. In medicine the practioner classifies a set of characteris- 
tics (a syndrome) and uses his know] edge of the effective treatments 
available to remedy the condition,, He then monitors the progress of 
the treatment® If all goes normally, the condition is correctedo If 
not, the practioner 1 modifies treatment (prognosis) and/or orders 
further tests to modify the classification (diagnosis) of the condition, 
and if necessary, calls in specialists to extend his knowledge 
resources, moves the patient to the hospital to extend his physical 
resources, etc 0 

There are, of course, dissimilarities between medical and 
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educational practice© Medical men deal general! y with short term 
problems of a clinical dysfunctional nature© The treatment range at 
their disposal tends to be drastic and when appropriate dramatic in 
its effectiveness © Educators, on the other hand, generally deal with 
long term developmental situations. The procedures available are less 
spectacular, slower acting, and much more complex© However, learning 
research is now providing increasingly powerful tools for the educator. 
Among these are CMI (Computer Monitored Instruction) , and CAT 
(Computer Assisted Instruction). These two procedures alone, along 
with the meaningful, interpretation of right and wrong answers in the 
terms just indicated might greatly extend the capabilities of education© 
The essential problem with the bright picture just painted is 
that at the moment, it contains too many unanswered "ifs." The next 
section spells out some of the research which might be conducted to 
help to make this dream become a reality© 

Furthe r Research Sug gested by the Findings of This Study 

There are several areas for further research suggested by this 
study© One of these involves the host of problems which an extension 
of test theory could generate© The solution to the "multiple 
interpretation" problem presented in this study, although suggested and 
strongly supported by the findings of this study, is probably only one 
of a range of possible solutions some of which may be more practical 
than others© Those individuals interested in mathematical statistics 
could pursue many avenues from this single problem© At a more practical 
level, there are a host of numerical analyses problems in the 
implementation of the particular procedure proposed in this study. 
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Subsequent, to the effective implication of an effective 
analytical procedure, there is the possibility of a host of studies 
into the characteristics of specific tests and classes of test, into 
the conditions under which nominal and ordinal categories form, into 
the types of relationship among categories which are normally found 
and the conditions under which these relationships occur Q Second, 
order factoring of the centroid matrices seems a logical first step 
but perhaps the entire structure could be integrated into a single 
analytic procedure and a single model 0 

Another essential area for research involves the formulation 
and resolution of problems arising from the interpretation of clusters 
after their statistical characteristics have been dete mined 0 Attend- 
ant to this problem is the cross-validation of interpretation to 
independent samples of equivalent or nearly equivalent profile 
characteristics* A profile in this context is a set of clusters and 
its attendant statistical and logi co-semantic characteristics© 

The formation of a generic model for a range of type of test 
opens the possibility for the computer generation of a test of partic- 
ular construct characteristics derived from the past performance 
characteristj.es of a large number of items in a pool of items© With an 
even larger pool of items, the computer could generate and administer 
a branching type programme tailored to a wide range of individual 
differences with the add of researchable adjustments to the test 
construction model© Tn this latter case, the computer could update its 
performance statistics on the item pool as students take the course, 
and in so doing refine its own course* 

Another area for research could be the reworking of many of the 


studies related to educational me thodol ogy , evaluating the methods with 
the profile analysis procedure suggested here© Problems of matching 
teacher, method, student, and programme could bo opened to detailed 
research, paving the way to the much broader use of diagnostic- 
prognostic practices in education than their present use G Studies into 
the relationships between achievement, personality, intellective and 
perhaps even genetic disposition variables whould also seem reasonably 
possible from these small beginnings 0 

Another area for research could be the precise definition of 
the developmental sequences through which children pass, the optimum 
ways of modifying these sequences toward specific goals and the degree 
to which these sequences can be modified. The charting of developmental 
patterns might lead to earlier and more precise identifications of 
specific talents© Also, the extension of the Guidelines to include the 
full range of academic performance might help to answer questions about 
the relative importance of content and of process in particular subject 
areas and for specific stages of development^ 

Finally, there is the psychological question as to whether 
intellectual development is continuous ana' cumulative, or discrete and 
taxonomic, or some combination of these two c In this latter case which 
aspects of intellectual development are continuous and which are 
discrete and how do they interact? Can critical phases and critical 
experiences be identified and matched so as to extend human 
capabilities? 

This list is not exhaustive 0 It is left to the reader to 
extend it himself in keeping w ith his own special interests© 
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APPENDIX A 

TABLES OF BASIC DATA 


1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 

21 

22 

23 

24 

25 

26 

27 

23 

29 

30 


PHI COEFFICIENTS FOR THE RIGHT ANSWERS 
FOR GROUP * A* 

LOWER TRIANGULAR MATRIX 



1 

2 

3 4 

5 6 

7 8 

9 10 11 12 

13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 


1000 * 

183 1000 
-115 -061 1000 
-005 050 -012 1C00 

C20 -104 047 -062 1000 


036 

072 

024 

091 

050 

1000 


















020 

053 

-082 

030 

027 

061 

100C 

















082 

146 

-005 

122 

-005 

-017 

049 

1000 
















-021 

-026 

-016 

093 

-137 

065 

047 

-090 1000 
















131 

094 

-012 

-053 

075 

-061 

084 

038 -072 

1000 















016 

-038 

128 

-053 

-060 

010 

021 

-013 092 

214 

1000 













H3 

-062 

115 

-018 

189 

Oil 

175 

047 

-038 050 

029 

-085 

1000 













107 

045 

125 

200 

046 

250 

071 

120 -063 

-2C5 

-066 

061 

1000 











1 1 

K 

024 

079 

149 

074 

164 

156 

115 

-059 -084 

074 

075 

027 

067 

1000 











163 

-055 

-029 

-018 

114 

-045 

187 

067 -207 


-087 

-029 

022 

051 1000 










<o 

032 

015 

038 

-025 

-009 

061 

039 

068 -034 

073 

087 

116 

055 

110 -143 

1000 










005 

019 

235 

007 

035 

172 

032 

229 077 

148 

-012 

Oil 

150 

103 -068 

072 

1000 









-C37 

-069 

-038 

105 

226 

-127 

123 

-078 -184 

-055 

-115 

039 

-C43 

010 525 

-252 

-035 1000 









016 

017 

038 

147 

128 

013 

036 

147 -000 

147 

112 

070 

041 

069 131 

-078 

034 050 

1000 








083 

030 

-077 

-030 

-017 

-054 

014 

037 105 

-068 

054 

226 

076 

C27 -043 

142 

-046 -008 

-059 1000 








006 

214 

-082 

-072 

-038 

073 

023 

-009 053 

091 

050 

112 

-036 

009 -019 

056 

025 -084 

169 063 

1000 







014 

101 

-144 

094 

113 

-181 

124 

125 001 

094 

-097 

-027 

-100 

-090 191 

-009 

022 434 

-066 038 -074 1000 







000 

040 

093 

039 

033 

-103 

097 

099 -080 

-077 

045 

001 

087 

-126 -072 

006 

032 -020 

-C97 -004 

-069 010 1000 







-093 

090 

-090 

-084 

121 

062 

-066 

-005 -047 

-030 

-021 

168 

-071 

091 C01 

066 

044 -123 

141 068 

196 -362 -056 

1000 






023 

047 

-007 

072 

-068 

118 

003 

082 030 

072 

055 

-007 

-118 

000 -055 

045 

-001 -133 

-079 045 

228 -026 -126 

203 

1000 





-033 

-033 

053 

-006 

006 

124 

-026 

-053 -032 

033 

114 

115 

048 

-071 -113 

111 

-120 -095 

051 -036 

-068 -104 003 

-016 

-148 1000 





058 

006 

053 

140 

-064 

202 

001 

053 160 

-020 

-041 

095 

088 

-054 -178 

-048 

124 -118 

022 -114 

077 -116 -C31 

170 

221 -165 

1000 




191 

227 

075 

-032 

022 

-053 

028 

228 -149 

047 

-066 

-082 

-231 

075 029 

135 

109 -092 

169 -034 

016 077 160 

057 

033 -137 

029 

1000 



073 

039 

012 

108 

062 

061 

-084 

-038 -052 

007 

053 

012 

-038 

002 -156 

123 

-078 -105 

017 -009 

031 -094 -001 

030 

119 -033 

020 

071 

1000 


-047 

-036 

095 

017 

010 

047 

-217 

-039 -042 

-069 

084 

-145 

095 

03*1 -010 

003 

-004 -064 

051 -140 

-001 -084 -147 

078 

-025 037 

155 

019 

026 

1000 


♦The numbers on this Table should be multiplied by 10 - . 


PHI COEFFICIENTS FOR THE RIGHT ANSWERS 
FOR GROUP ‘S' 

LWEH TRIANGULAR MATRIX 


1 100G* 

2 187 1000 

3 C06 -025 1000 

4 -070 -003 -167 1000 

5 -072 -023 -039 -051 1000 

6 -006 007 098 125 025 1000 

7 117 -088 -026 -005 -010 -072 1C00 

8 -060 -064 -111 -039 -030 -010 027 1000 


9 

-050 -081 

221 

-005 

014 

-024 

-065 

143 

1000 



















10 

016 003 

093 

063 

132 

014 

-036 

186 

171 

1000 


















11 

009 -168 

-014 

-022 -084 

057 

-Oil 

-029 

021 

-017 

1000 

















12 

113 144 

145 

-270 

170 

085 

040 

-013 

-086 

071 

-058 

1000 
















13 

030 -043 

007 

026 

222 

086 

-033 

-037 

025 

121 

031 

158 

1000 















14 

049 090 

180 

214 

ocb 

015 

-017 

C41 

153 

107 

001 

073 

119 

1000 














15 

123 103 

-073 

-008 

-039 

063 

136 

007 

-187 

009 

-090 

013 

184 

-139 

1000 













16 

060 -036 

0S9 

036 

155 

111 

-160 

007 

084 

042 

-055 

028 

-093 

095 

-051 

1000 












17 

042 -004 

281 

-015 

028 

-049 

182 

-080 

046 

016 

-077 

024 

-100 

135 

-083 

060 

1000 











18 

061 -013 

064 

-081 -099 

-089 

330 

-118 

-280 

-073 

072 

-020 

149 

-084 

387 

-314 

190 

1000 










19 

065 -005 

033 

130 

-058 

037 

129 

010 

-006 

020 

051 

065 

107 

182 

010 

-075 

153 

-023 1000 










20 

-114 -094 

027 

-015 

-109 

-045 

128 

-050 

012 

03C 

110 

015 

-128 

103 

-015 

-096 

053 

161 -159 

1000 









21 

-156 -126 

123 

-141 

-041 

-057 

-099 

-002 

152 

-Oil 

-041 

-051 

113 

152 

-061 

-019 

031 

-013 042 

123 

1000 








22 

-015 225 

-121 

104 

-015 

-151 

080 

-116 

-231 

- 11 ? 

-m 

-047 

-068 

033 

232 

005 

072 

304 -051 

069 

-043 1000 








23 

-006 -057 

005 

-026 

093 

072 

- C27 

-010 

041 

014 

-089 

-041 

-090 

-072 

-074 

042 

037 

-069 063 

-045 

006 -209 

1000 







24 

030 111 

099 

-140 

131 

069 

-234 

-043 

130 

052 

056 

173 

-005 

057 

-121 

079 

-033 

-307 037 

-034 

-062 -320 

279 

1000 






25 

037 104 

098 

050 

-008 

-073 

017 

-010 

269 

014 

-059 

085 

-090 

-014 

-074 

042 

037 

-175 -151 

101 

038 - 0 33 

101 

-015 

1000 





26 

-023 -071 

-074 

154 

-046 

060 

-053 

-160 

- 1G9 

-026 

086 

-040 

072 

041 

061 

021 

066 

-040 046 

127 

038 -016 

-101 

039 

-071 

1000 




27 

-Oil -032 

017 

104 

006 

203 

-165 

085 

160 

119 

-009 

071 

022 

010 

-064 

096 

-223 

-335 -191 

-016 

063 -296 

096 

258 

203 

-092 

1000 



28 

078 052 

-111 

-007 

102 

059 

-010 

078 

-024 

-055 

-084 

-013 

- C84 

008 

-039 

155 

123 

001 -053 

060 

107 121 

059 

-064 

025 

059 

-035 

1000 


29 

060 040 

089 

-23 2 

-124 

-198 

049 

-066 

123 

-054 

048 

102 

on 

164 

-051 

-139 

060 

-007 073 

040 

019 006 

— 06l 

179 

042 

-122 

-073 

-084 

1000 

30 

-005 -020 

029 

-071 -025 

—063 

-123 

059 

147 

125 

152 

081 

042 

168 

-060 

210 

-005 

-223 -063 

033 

150 -145 

078 

146 

113 

-041 

073 

-025 

163 10C0 


* theso nunbers should be nultiplied by 10“" 


MS 
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TABLES 34, 3.5 Sc % 


1 


GIVE THE 

PHI COEFFICIENTS FOR THE WRONG ANSWERS 


FOR GROUP A 


LOWER TRIANGULAR MATRIX 


IN THREE PARTS 


10 11 12 13 


15 16 17 18 19 20 21 22 23 


28 29 30 31 32 3? 


36 37 3o 39 40 


1 1000 * 

2 ?41 1000 

3 -271 -022 1000 

4 -173 022 -040 10OO 


5 

-16J 

049 

-028 

-077 

10C0 





















6 

-067 

-085 

-018 

115 

192 

1000 




















7 

-122 

-005 

301 

-005 

132 

-029 

1000 



















8 

049 

122 

-018 

-051 

131 

027 

-069 

1000 


















9 

-032 

039 

153 

-116 

026 

108 

057 

158 

1000 

















10 

-049 

059 

-053 

057 

039 

163 

-101 

132 

099 

1000 
















11 

079 

032 

090 

-049 

048 

-075 

-001 

-004 

114 

-002 

1000 















12 

C31 

03? 

090 

-049 

130 

-075 

-001 

032 

114 

085 

-118 

1000 














13 

-109 

096 

403 

-040 

-02o 

-01c 

-023 

-018 

153 -063 

090 

-061 

lOOu 













14 

-06’ 

017 

-018 

-050 

-034 

-022 

-029 

-073 

-052 

-078 -075 

049 

-013 

1000 












19 

06b 

118 

-018 

-050 

192 

319 

-029 

027 

108 

042 

-075 

-075 

-018 

-022 

1000 











16 

054 

-02 c 

-015 

-040 

-028 

-018 

-023 

-018 

-042 

083 -051 

090 

-015 

-013 

-018 1000 











17 

018 

-032 

-0?0 

-130 

116 

075 

0S7 

331 

001 

C45 

-016 

-016 

-241 

-049 

C75 -241 

1000 










1C 

030 

-056 

-042 

038 

-080 

-052 

-067 

-077 

-121 

-070 

-001 

-117 

-042 

267 

108 153 

-172 

1000 









19 

-139 

-070 

-037 

152 

-071 

3' ’7 

-059 

005 

-024 

-037 

-154 

-027 

-0J7 

-C46 

-046 -037 

091 

-107 

1000 








20 

-122 

-163 

625 

-065 

-044 

-029 

170 

008 

182 

-101 

-OOl 

096 

301 

-029 

-029 -023 

C01 

-067 

216 

1000 







21 

-026 

051 

-034 

092 

-C64 

150 

-054 

126 

-007 

125 

-001 

-070 

-034 

-041 

-041 -C34 

070 

-007 

014 

-054 

1000 






22 

066 

-I06 

-OU 

-C50 

-U34 

319 

-029 

077 

108 -078 

-075 

049 

-018 

-022 

-C22 -018 

075 

-052 

483 

503 

-041 

1000 





23 

127 

032 

-061 

Oil 

-034 

049 

-097 

032 

-001 

085 

061 

-073 

-061 

049 

172 - 061 

163 

114 

-091 

-C?7 

138 

-075 

1000 




24 

-109 

-028 

4^3 

-040 

24o 

-01c 

301 

-oie 

-042 -063 

090 

241 

-015 

-016 

-018 -015 

061 

-042 

-037 

301 

-034 

-018 

-061 

1000 



25 

112 

098 

-''67 

-074 

026 

-083 

-108 

000 

024 

036 

141 

-069 

-067 

033 

-4583 217 

-015 

024 

009 

-017 

-024 

033 

-027 

-067 

1000 


26 

026 

-076 

-051 

126 

-097 

-063 

-081 

-004 

-017 

-075 

139 

-062 

-051 

076 

-063 -051 

-139 

112 

-050 

-061 

-117 

-063 

039 

-051 

048 

1000 

27 

-155 

-128 

-021 

-05o 

157 

-026 

l‘)8 

061 

079 

-09C 

021 

021 

-021 

-026 

-026 - 021 

086 

079 

-053 

-033 

-0*:8 

-026 

-086 

3-41 

-096 

-073 1000 

^b 

169 

190 

-070 

— 0?9 

-134 

■OuO 

066 

029 

0x2 -C6J 

039 

-126 

-070 

142 

-Oco 069 

-C39 

012 

-119 

-112 

-033 

-Oj6 

0-1 

-070 

1-3 

-060 0.8 1000 


29 039 -040 -035 -010 -Ooo -044 -C57 -OU -01c 105 -014 -014 -C;5 -044 -044 412 -119 242 -O'.'O -057 022 -044 052 -035 024 100 109 014 1000 

30 030 -107 -Oio 254 -020 -013 -016 -099 -C3C 162 -043 169 -O10 -013 -013 -010 -169 -030 -026 -016 -024 -013 -043 -010 -047 202 -015 -049 290 1000 

31 -434 -064 -0..3 -065 13- -02 -037 010 057 -007 -00; 192 -023 237 -029 -023 097 -067 -059 -037 -054 -029 -097 -023 -017 026 -033 -112 -057 -016 1000 

32 01C -403 lev -025 -074 -04. 07i -024 -032 -169 -038 065 -039 -048 -048 16? 030 207 077 336 -089 292 023 169 -005 002 093 041 180 -027 -C62 1000 

33 -07. -055 -C66 101 -048 036 -014 15* -025 045 -01 -019 -066 -0*1 -081 -066 -151 030 C14 -014 178 -081 024 -Oc6 -144 056 110 037 093 156 -OU 000 1000 

34 -191 -107 7 05 -028 -020 -013 4 41 -099 -030 -045 -043 169 -010 -013 -013 -010 043 -030 -025 -441 -024 -013 -043 703 -047 -03 6 -015 -049 -025 -007 -016 265 -047 1000 


35 

161 

-090 

-028 

022 

-289 

-085 

-084 

062 

039 

-046 

-041 

-005 

-020 

-085 

C17 

096 

078 

067 

035 

075 

051 

118 

032 

-028 

-OC6 

007 -039 

088 

-040 

-107 

-064 

053 -090 

068 

1000 









36 

099 

-105 

080 

-007 

-044 

-OU) 

082 

-064 

092 

013 

-011 

-OU 

-065 

-080 

-080 080 

-033 

-075 

020 

-OU 

-082 

-080 -097 

080 

270 

U2 

OU 

127 

-029 

159 

-103 

055 

-047 

159 

10 . 

1000 








37 

-037 

054 

C50 

-017 

059 

008 

-129 

C08 

068 

-OU 

052 

130 

050 

006 

003 

131 

-246 

169 

-039 

038 

-066 

ooe 

-064 

050 

-Ool 

-022 

071 

07 0 

034 

127 

121 

052 

078 

-057 

-041 

015 

1000 







38 

-109 

-028 

493 

-040 

-028 

-018 

301 

-141 

153 -063 

-061 

241 

-015 

-015 

-013 -015 

061 

-042 

-037 

301 

:oo 

-016 

090 

493 

O'*? 

-051 

-021 

-070 

-035 

-010 

-023 

169 

-066 

705 

096 

225 

-OU 

1000 






39 

—062 

-122 

-054 

108 

-103 

067 

OU 

-244 

-155 

049 

114 

-127 

-054 

-066 

-066 -054 

-056 

032 

OCi 

-O06 

026 

-06 6 

114 

-054 

025 

137 

-077 

009 

-131 

-033 

-O-.’ 6 

-010 

216 

-038 

-1C1 

-C53 

-0*9 

-0,4 

1000 





40 

027 

070 

230 

-0.3 

-057 

-037 

-048 

-100 

113 

-130 

107 

-124 

230 

176 

-037 -030 

-184 

U3 

-076 

US 

051 

-037 

-047 

-030 

-065 

155 

-043 

212 

042 

-021 

-048 

027 

-062 

-021 

007 

016 

035 

-030 

-027 

1000 




41 

-002 

-086 

-080 

040 

-030 

US 

125 

090 

024 

074 

-330 

060 

-080 

010 

-098 052 

018 

-027 

0 77 

041 

181 

226 

021 

-OU) 

-073 

-052 

-020 

-021 

097 

129 

-043 

056 

087 

-056 

042 

024 

-135 

-OU) 

-0.2 

039 

IOCO 



42 

-077 

-03C 

-0*9 

003 

-032 

098 

019 

-09* 

-014 

089 

-U3 

-371 

-069 

-006 

093 

037 

-042 

22Q 

202 

019 

-031 

096 

OU) 

-069 

-022 

067 

053 

-081 

116 

-063 

-143 

073 

133 

-063 

001 

003 

060 

-Oo9 

077 

012 

105 

1000 


43 

-101 

-064 

200 

092 

-064 

-041 

096 

126 

172 

-OU 

-070 

068 

-034 

-041 

-041 

200 

-06 d 

Oo3 

014 

245 

-078 

150 -070 

200 

-024 

-040 

-04 0 

-033 

022 

-024 

-054 

197 

-020 

306 

051 

165 

235 

200 

-124 

-069 

059 

02* 

1000 

44 

-04 •! 

-028 

160 

047 

-077 

-050 

064 

-ICO 

038 

-176 

190 

-049 

361 

-050 

-050 -040 

-070 

-116 

-1C 3 

064 

-093 

-050 

-168 

-040 

150 

059 

-050 

-029 

-090 

-028 

064 

-025 

-069 

-02o 

-077 

050 

-017 

-040 

-0*5 

225 

-064 

-147 

092 1 CO 

45 

-OOl 

097 

-026 

-071 

113 

-032 

149 

-032 

-074 

-111 

-018 

-107 

-026 

-032 

-032 -026 

-158 

-07 -J 

-065 

-041 

-059 

-032 

-013 

-026 

048 

-090 

•037 

121 

069 

-Olo 

-041 

-063 

052 

-018 

-121 

-029 

OU 

-026 

C01 

-052 

-C63 

066 

07 0 -071 1000 


• thone nuacera ahouixl bo cmltlpllcd by 10”' 




TABLE 'j>h 




1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

15 16 

17 

18 

19 

20 

21 

22 

23 

24 

25 

26 

27 

28 

29 

30 

31 

32 

3J 

‘ 34 

35 

36 

37 38 

l 39 

1 40 41 42 

43 

44 45 

46 

- 067 * 

-186 

-018 

115 

-034 

319 

-029 

027 

-052 

042 

-075 

-075 

-013 -022 

-022 -018 

075 

-052 

131 -029 

150 -022 

049 

-018 

-083 

-063 -026 

028 

-044 

-013 

-029 

12 2 

154 -013 

-186 

039 -099 -018 

i 200 -037 

110 -006 

-041 - C50 -032 

47 

-039 

049 -021 

-058 -040 -026 

-033 

061 

218 

119 -086 

123 

-021 

-026 

-026 - 021 

-343 

079 -053 

-033 

-048 

-026 

-086 

-021 

106 

048 -030 

-100 -050 

-015 

-033 

-05 5 

008 -015 

-039 

on 

-022 -021 

039 

-043 

-020 -037 

n9 

ce5 -037 

48 

093 

094 

-295 

-067 

094 

-080 

-031 

063 

010 

016 

001 

053 

-123 

-221 -080 -123 

203 

-848 

053 

-141 -045 

061 

-050 

-123 

-012 

-056 

-052 

002 

-m 

035 

079 

—22 0 -OIS 

1 -208 

-032 

-027 

-169 -223 -033 

-163 

002 -127 

-124 

C01 

. 087 

49 

-048 

-027 

128 

006 

004 

085 

148 

003 

064 

-055 

-095 

-095 

7-28 -059 

085 -048 

-166 

064 

-122 

143 

-111 

-059 

061 

12ft 

-124 

008 

057 

203 

-039 

- 0J4 

036 

016 

180 214 

059 

038 

096 

■ 128 

040 

032 

012 101 

051 -064 

. C19 

50 

086 

-005 

-023 

—065 

-044 

-029 

-037 

010 

057 

180 

-001 

096 

-023 

-029 -029 625 

-096 

C57 

-059 -037 

-054 

- C29 

096 

-023 

074 

-081 -033 

066 

230 -016 

-037 

203 

-106 

-016 

154 

-on 

033 

1 -023 -086 

- C4S 

041 

. 019 

096 -065 

-041 

51 

- C56 

C06 

031 

087 

-042 

-OU 

010 

032 

-029 -008 

019 

131 

031 

-115 

090 - 093 

055 

-029 

133 

170 

-215 

192 

-203 

Oil 

-082 

047 

045 

032 

-116 

no 

010 

109 

036 

-066 

128 

-128 

094 

-093 

056 

000 -020 

1 -073 

015 

-061 -091 

52 

-056 

-057 

-030 

020 

-057 

-037 

-048 

-038 

-086 

-130 

030 

-124 

-030 

176 

-037 -030 

-184 

213 

034 

-04 8 

051 -037 

-047 

-030 

007 

-018 

-043 

141 

C42 

-021 

-048 

13 3 

012 

-021 

-183 

-132 

035 

-030 -027 

072 

033 

1 141 -069 

020 

1 100 

53 

050 

013 

-055 

100 

069 -068 

014 

046 -098 

-008 

-040 

008 

-055 

—OGb 

-068 -055 

-055 

025 

-073 

-088 

094 

-068 

-2 29 

-055 

-165 

-033 

-079 

-221 

-063 

-039 

- O08 

-081 

157 

-039 

052 

-016 

106 -055 

-101 

-031 -010 -058 

C91 

-026 

- 097 . 

54 

-176 

-121 

-034 

092 

06 ? 

-041 

-054 

014 

-007 

-078 -070 

-OOl -034 

-041 

-041 200 

-068 

-097 

212 

245 

030 

342 

-070 -034 

172 

-040 

119 

095 

125 

-024 

-054 

1?7 

046 

-024 

051 

lie 

115 

-034 

-124 

-069 

059 

028 

030 -001 

era 

55 

-075 

-166 

0*33 

077 

-030 

052 

003 

091 

0C5 

-125 

-020 

025 

093 

-073 

-073 -059 

-071 

064 

-086 

ICO 

074 

052 -065 

093 

-?74 

-055 

103 

-117 

- 0C9 

-042 

CCJ 

155 

293 

-042 

-131 

-132 

143 -059 

026 

-043 

1 13 

018 

C74 

017 

-104 

'56 

-009 

039 

-066 

064 

-095 

-002 

025 

-094 

-151 

032 

- OIL ’ 

020 

041 

-106 

-002 041 

-020 

- 1C2 

159 -056 

036 

-002 

-056 

- C86 

-104 

-002 

-123 

-064 

- C41 

-061 

025 

032 -030 

-061 

039 

-164 

114 

-086 

-073 

-111 

060 

205 

036 

064 

-C 02 

57 

077 

003 

-039 -025 

-074 

-048 

071 

076 

-032 

012 

023 

-100 -039 

-048 

-043 -039 

038 -032 

165 

-062 

-089 

-048 

- 0J8 

-039 

-063 

002 -055 

- 0 ? 6 

-094 

-027 

- C62 

067 

118 

-027 

-099 

125 

052 

-039 

189 -073 

-133 

022 

006 

-107 

-068 

53 

053 

-060 

048 

029 

055 

112 

035 

021 

-137 

16S -072 

-033 

- 0C2 

005 

112 04 " 

185 

013 

-099 

055 

230 

- 1C1 

in 

048 

-162 

- 15u 

-117 

-095 

-028 

-058 -4340 

-006 -005 

125 

-060 

-142 

-087 

17 0 

-012 

-166 

-in 

C48 

-oie 

-125 

-145 

59 

110 

070 -030 

123 

-057 

-037 

-043 

087 

014 

021 

03C 

107 

-030 

-037 

-037 -ojo 

047 

014 

-076 

-048 

-069 

-037 

-047 

-030 -065 

-010 

-<-'43 

-001 

-072 

-021 

-048 

027 

012 

-021 

070 

016 

035 

-030 

056 

-06 \ 

-026 

-182 

170 - Od3 

-052 

60 

-028 

-035 

- C51 

-141 

- C05 

- O63 

026 

-004 

043 

023 

-012 

009 

11c 

214 

-063 -051 

-009 

-017 

-05 8 -Obi 

03L 

-063 

-012 

-051 

001 

-066 

-073 

033 

-124 

-036 

134 

002 -088 

.036 

-076 

064 

-065 

118 -080 -018 -015 

-143 

-040 

-008 

207 

61 

-334 

-085 

-018 

115 

-034 

-022 

-029 

-073 

-052 

042 -075 

-075 

-018 

- G?2 

-022 -018 -045 

-052 

131 

-029 

-041 

-On 

-075 

-018 

-003 

-063 

271 

142 

-044 

-013 

-029 

-04 3 

006 

- 0*3 

017 -080 

U5 

-010 

067 

-037 

-098 

093 -041 -050 -002 

62 

-001 

-266 

-026 

-071 

-049 

212 

-041 

-104 

-074 

-025 

-107 

-107 

-026 

-032 

- 03 ? -026 

107 

040 

061 

149 

070 

212 

070 

-026 

-035 

009 

-037 

-123 

-062 

- 01G 

-041 

-068 

052 

-018 

C24 

056 

Oil -026 

096 

-052 

014 

288 -059 

047 

-045 

63 

- 02 ? 

-056 

-101 

-036 

076 

077 

-004 

-024 

-101 

060 

055 

-127 

022 

170 

-023 022 

200 

007 

004 

-161 

-119 

-023 

055 

-101 

-010 

057 

-144 

-113 

-190 -071 

075 

-067 

-456 

-071 

064 

010 

-115 

-101 -096 

045 

-007 

-033 

C51 

155 

-105 

64 

153 

028 

-193 

-536 -076 

-017 

-137 

079 

062 

133 

-006 

114 

-059 

093 

093 075 

006 

062 

021 

-137 

-136 

093 

034 

-039 

1:3 

-050 

1C7 

067 

005 

-106 

035 

-019 

-039 

-135 

-038 

065 

0^2 

-059 

063 

-121 

-042 

092 

-012 

-no 

-104 

65 

- Ob2 

CIO 

- 0-44 

-046 

-083 

-054 

-070 

-055 

-125 

-080 

-069 

-069 

-044 

102 

-054 -044 

-100 

093 

-031 

-070 

-013 

-054 

-069 

44 

Oil 

-026 

C73 

050 

145 

- C31 

051 

C39 

070 - C31 -453 

-031 

-046 

-044 

082 

008 

-041 - O0C -013 

104 

145 

66 

-076 

050 

077 

-069 

-048 

-OLl 

-014 

009 

030 

045 

02J 

024 

077 

- 0-1 

-081 -066 

062 

250 

014 

170 

C46 

1.4 

-019 

-066 

-084 

-040 

110 

- 12<J 

156 

-047 

071 . 

118 

066 

-047 

015 

-293 

ns 

-065 

-060 

-OS 2 

-025 

133 

046 

044 

052 

67 

Ob 8 

075 

-123 

08 b 

-038 

-053 

-197 

03 3 

-077 

059 

-Oil 

-Oil 

-123 

046 

046 -123 

190 

-077 

04 5 

-120 

050 

046 

Ocl 

-123 

10O 

U 2 

-176 

-062 

-036 

-007 

-040 

-131 

-060 -087 

047 

040 

-684 

-003 

087 

056 

090 

- C91 

-173 

-OCT 

- C75 

68 

039 

056 

-044 

029 

-033 

-054 

-070 

-420 

-125 

030 

-125 

044 

-044 

102 

-054 -044 

-100 

020 

050 

-070 

-100 

- 0.-4 

044 

-044 

Oil 

-026 

-062 

050 

-106 

-031 

-070 

-038 

-144 

-031 

-083 -031 

-046 

-044 

021 

ooe 

0C3 

111 

-100 

-121 

- or / 

69 

077 

049 

-021 

-058 

-040 -026 

-033 

148 

-060 -090 

- O06 

021 

-021 

-026 

-026 -021 

086 

079 

254 

198 

-04 e 

271 

021 

-021 

C05 

-073 

-030 

098 

109 

-015 

-033 

093 

008 

-015 

049 

i'll 

-315 

-021 

-077 

142 

074 

-037 

-048 

ce5 

-037 

70 

107 

-032 

-135 

-084 

-058 

-<166 

09 - 

-009 

-154 

-585 

018 

-018 

-013 

"066 

034 - 013 

127 

079 

013 

013 

-142 

133 

-018 

-135 

127 

-026 

068 

120 

-005 

-095 

019 

I89 _ 09 * 

-095 

116 • 

-no 

037 

-135 

-107 

-275 

023 

-035 -030 

060 

n9 

71 

-067 

1.18 

-013 

115 

192 

319 

-029 

-073 

108 

163 

-075 

-075 

398 -022 

3!9 -° 16 

-052 

131 

-029 

-041 

-022 

049 

- 01 * 

-003 

-063 

-026 

-006 

-044 

- 01J 

-029 

-048 

-081 

-013 

017 • 

-086 

008 

-010 

- 06t 

-037 

-098 

098 -041 

115 

-032 

72 

020 

-126 

-040 

047 

-077 

115 

064 

-100 

038 -001 

Oil 

-168 

-040 

- 050-050 -040 -070 

038 -OIL 

-065 

-001 -050 

-049 

-040 

038 

059 

085 

OLl 

-098 

-028 

-065 

057 -013 

-028 

120 

165 

-121 

-040 

044 

123 

-012 

-247 

-001 

326 

-071 

'73 

112 

09b 

-007 

-OIL 

-051 

- O03 

-017 

000 

-140 

077 

-027 

015 

-067 

033 

033 -067 

027 

133 

069 

-108 

041 -033 

- CL7 

-175 

-113 

040 

106 

-207 

-164 

-047 

074 

- 1*1 

-024 

-047 

028 

067 

029 

-067 

-021 

077 

-07 3 

119 ■ 

-155 

-074 

048 

74 

050 

-ocr 

024 

-031 

-055 

- O ' 5 : 

078 

082 

-049 

068 

-047 

O63 

-099 

-122 

-021 147 

010 

-002 

113 

0-/8 

-172 

131 

-003 

024 

-010 

022 

122 

1S9 

194 

104 

-159 

240 

004 

104 

203 

265 

-010 

024 

-050 

-077 

034 

069 

169 • 

-275 

-000 

75 

143 

-102 

-146 

-029 

-122 

-257 

-051 

Oin 

— 09 5 

- 03C 

-044 

1C1 

-145 

054 

-412 -146 

100 -093 

031 

-051 

013 

054 

-156 

■144 

042 

089 

-073 

053 

-145 

031 

070 

008 

-017 

C31 

129 

139 

-002 

OU 

- C02 

-008 

090 

-in -074 

048 

-509 

76 

-093 

-13 o 

123 

067 

-094 

-061 

-079 

020 

-010 

-066 

-104 

ici 

123 

221 

-061 -050 

053 

056 

093 

251 

-035 

221 

-155 

-050 

-004 

-053 -071 

-143 

-044 

-035 

141 

2:0 - C78 

-035 

115 • 

-121 

169 

-050 

-127 

-ou 

132 

- 0C2 

045 

067 

-027 

77 

- C56 

003 

169 

C57 

-074 

-04 e 

-062 

-224 

-032 

-048 

085 

-033 

376 

122 

-049 376 

-640 

207 

-099 

071 

006 -043 

-ICO -039 -063 

140 -055 

093 

180 

265 

-062 

-OJo 

235 

-027 -040 

125 

266 

-039 

056 

239 

-050 

125 

102 

057 

173 

78 

- 0s7 

-085 

398 

-050 

-OJl 

-022 

237 

-073 

108 

042 

-075 

172 

-018 

-022 

-022 -018 

-049 

-052 

-046 

237 

-041 

-022 

-075 

J98 

033 

-063 

-026 

020 

-044 

-013 

-029 

122 

-081 

573 

017 

277 

006 

396 

-066 

-037 

010 

-no 

150 

115 

- Cj2 

79 

346 

1?6 

-113 

-122 

-018 

-139 

-104 

134 

-047 

174 

1*0 

104 

- U3 

059 

059 129 -033 

-047 

-288 

-181 

074 

-139 -003 

OOG 

053 

-033 

ou 

154 

153 

091 

-104 

-033 

-067 

-080 

007 -017 -003 

-113 

-068 

o?e 

043 

-140 

Old -026 

024 

80 

077 

-092 

-096 

028 

-049 

-017 

005 

-032 

056 

-095 

-032 

041 

023 

-017 

-118 -096 -041 

008 

070 

-154 

007 

-017 

041 -096 

041 

076 

-049 

-054 

-070 

-068 

004 -OCJ 

-015 

-068 -092 

105 ■ 

-150 

C28 

042 

-070 

054 

092 ■ 

-165 

077 

-097 

31 

060 

085 

077 

-069 

-048 

-081 

170 

-118 

-025 

-038 -062 

-062 

-066 

-081 

-081 221 

-151 

030 

-107 

-014 

-152 

-081 

024 

077 

217 

056 

no 

-002 

156 

-047 

-106 

039 -055 

156 -090 

241 

041 

077 

032 

012 

050 

062 

046 -013 

138 

82 

125 

180 

-279 

-117 

-080 

-207 

-002 

066 

-040 

132 

-175 

-020 

-113 

-071 

064 052 

274 

-103 

-218 

-340 

-032 

-343 

169 

-279 

057 

-037 

-043 

-021 

-019 

-196 

084 

-IS 9 

-044 

-196 

180 -052 -094 

-113 

007 ■ 

-147 -058 

-051 • 

-lOo 

014 

-199 

83 

-434 

-005 

301 

192 

-044 

237 

170 

-069 

-067 

180 -097 

192 

-023 

-029 

-029 -023 

097 

-067 

215 

170 

096 

-029 

-097 

?01 

-100 

-081 -033 

—112 -057 

-016 

378 

071 

078 

441 -005 -on 

C36 

301 

018 -048 

041 

-062 

06 

064 

-041 

84 

027 

-057 

-030 

020 

-057 

176 

-048 

025 

014 

-055 

030 - C47 

-030 

176 

176 -030 -030 

014 . 

034 

118 

051 -037 

-047 

-030 -065 

-013 -043 

070 

-072 

-021 

-048 

027 

085 

-021 -057 - 05d -032 -030 

056 

C72 -028 

141 

051 -003 

252 

85 

-334 

-186 

398 

115 

-034 

319 

237 

027 

-052 

042 

-075 

049 

-018 

-022 

-022 -018 

075 

-052 

131 

237 

150 

-022 

-075 

398 -003 

-063 

271 

028 

-■044 

-013 

-029 

122 

036 

573 

017 

039 

000 

398 

067 -037 

010 

-006 

150 -050 -032 

86 

-117 

-064 

133 

-058 

009 

-057 

-075 

-016 

142 

058 

127 

-033 

133 

090 -057 -047 

-020 

073 

-042 

271 

141 

090 

020 

-047 

087 

-163 

062 

-125 

046 

-033 

041 

023 

044 

-003 -044 

051 -073 

133 

001 -003 -067 

-060 -024 

013 

024 

37 

-039 

049 

-021 

085 

-040 

271 

-033 

061 

079 

119 

128 

021 

-021 

-026 

-026 -021 

-021 

_ C60 

100 -033 

119 

-026 

-086 

-021 

106 

-073 -030 

098 -050 -015 

-033 

-055 

213 

-015 -039 

U5 

071 

-021 

039 -043 

074 

-037 

119 -058 

175 

88 

-055 

007 

230 

020 

084 

-037 

-04 e 

150 

014 

-055 

261 

030 

230 -037 

-037 -030 -030 

113 

034 

113 

-069 

-037 

-047 

230 -065 

060 

142 

-143 

156 

-021 

-048 

133 

012 

-021 

007 - 

053 

035 

-030 ■ 

-no 

072 - 09 ? 

C12 

051 

123 

-052 

89 

-200 

-186 

398 

113 

-034 

319 

237 

027 

-052 

042 

-075 

049 

-018 -022 

-022 -018 

075 

-052 

131 

237 

150 - 02 . 

-075 

398 -083 -063 -026 -086 

-044 

-013 

-029 

122 

154 

573 

017 

039 

008 

398 

067 -037 

118 

-006 

150 -050 - 03 ? 

90 

-173 

-07 T 

-040 

126 

142 

115 

-065 

-003 

-116 -059 

130 

-108 

-040 

-050 

-050 - 04C 

049 -039 

067 

-065 

-001 

-050 

-049 

160 -074 

-074 

228 

026 

079 

-028 -065 

07 ! 

044 

-028 -077 

050 -017 

-040 

1C8 -083 -064 

-047 -OOl 

047 

-071 
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TABLE 35 


4<S 47 43 49 50 51 52 53 54 55 56 57 55 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 75 77 7 8 7 9 80 61 82 83 34 85 86 87 85 89 90 


4 $ 1000 * 

47 -026 1000 

4 ? -080 -052 1000 

49 085 -068 -076 1000 

50 -029 -033 -031 -077 1000 

51 -013 045 063 -005 -149 1000 

52 -037 -043 -163 -098 -045 -063 1000 

53 -065 -079 025 -126 -088 119 -031 1000 

$4 -041 -048 114 -111 096 130 051 020 1000 

55 052 024 -058 016 -095 184 113 112 004 1000 

55 103 -123 C75 -063 107 -081 150 035 -081 -005 1000 


37 

-04 C -035 

C61 

-123 

071 

1C9 

027 

049 

006 

279 

-125 

1000 

































33 

112 • 

-117 • 

-029 

-045 

035 

-047 

-035 

221 

-130 

037 

069 

-059 

1000 
































59 

-037 

-043 

013 

082 

- 04 " 

192 

-061 

031 -069 

113 

-111 

027 

-035 

1000 































60 

076 

043 

001 

066 

026 

-077 

068 

-086 

-117 

-055 

038 

-067 

015 

-018 

1000 






























61 

-022 

-026 

061 

-059 

-029 

192 

176 -058 

150 

052 

-002 

122 

-101 

-037 

-063 

1000 





























62 

212 

-037 

-014 

019 

-014 

-018 

-052 

-097 

078 

164 

-077 

-068 

160 

-052 

009 -032 

1000 




























63 

-023 

-056 

008 

- C76 

075 

-070 

045 

-110 

-119 

-114 

022 

-017 

124 

108 

098 -023 

111 

1000 



























64 

-017 

107 

018 

-031 

121 

-080 

086 

-096 -074 

-058 -061 

091 

-023 

017 

085 

- or > 

054 

032 

1000 


























65 

-054 

-062 

-109 

-012 

-070 

-092 

395 

-045 

-013 

108 

C7G 

039 

-052 

104 

226 

102 

-077 

020 

075 

1000 

























66 

-081 

110 

-214 

-118 

-105 

107 

085 

-069 

046 

034 

151 

000 

032 

012 

cob 

036 

-032 

-074 

037 

017 

1000 























I 3 

67 

-152 

-004 

132 

-155 

-043 

-046 

-067 

028 -062 

-211 

-183 

-032 

047 

-005 

051 

-053 

-075 

055 

-035 

032 

-114 

1000 






















>■ 

bd 

68 

102 

-062 

C19 

—012 

051 

-092 

104 

-045 

-100 -063 

171 

-038 

045 

-009 

-026 

102 

034 

-026 

075 

-059 

-090 

082 

10O0 





















' — 1 


-026 

0 

0 

-052 

057 

-033 

045 

-043 

-079 

119 

-085 -032 

093 

-117 

-043 

-073 

-026 

-037 

-055 

012 

073 

no 

168 

-062 

1000 




















70 

-066 

-105 

045 

020 

-CSC 

095 

097 

- C50 

082 

075 

091 

-010 

-198 

- 08 ? 

-066 

034 

-023 

012 

-013 

052 

009 -067 

C06 

063 

10C0 



















'-0 

71 

-022 

-026 

061 

229 

-029 

-013 

-037 

-068 

-041 

-073 

103 

-046 

0C5 

-037 

076 

319 

-032 

178 

-017 

-054 

-081 

-053 

102 

*> 

N 

-066 

1000 


















C \ 

72 

11 ? 

085 

-135 

006 

064 

038 -083 - C2o 

092 

-043 

-138 

139 

-022 

020 

-074 

-030 

-071 

061 

-057 

-046 

-126 

041 

-046 -058 

-084 

-050 

lOOO 


















73 

- 0C3 

CO 5 - 06C 

^■074 

074 

-047 

007 

05 « 

-155 

-060 

03C 

ill 

020 

007 

142 

033 

-035 

0C5 

160 

117 

137 

039 

11T 

005 

-130 

033 

-131 1000 

















74 

-021 

-054 

002 

-071 

156 

153 

-OV 

052 

225 

-070 -091 

038 

-118 

- Oi4 

-101 

181 

-102 

001 

121 

-020 

-101 

041 

-056 

034 

029 

-021 

-080 

-010 

1000 
















75 

- 1C2 

062 

173 

012 

-172 

0“>4 

077 

117 

076 

-082 

-070 

098 

130 

062 

- 05 ? 

-054 

077 

317 

076 

-082 

-070 

090 

130 

052 

-052 

-102 

-029 

042 

C66 

1C00 















76 

-061 

052 

-066 

-044 

-079 

107 

- C13 

064 

045 

110 

140 

0C9 

074 

-013 

-OOl 

oCc 

-007 

033 

-109 

-019 

165 

-009 

045 

052 

12C 

080 

-001 

012 

-CO 2 

00-4 

1000 














77 

-042 

-055 

-220 

ISO 

203 

-044 

239 

049 

102 

0«2 

-021 

067 

-059 

-079 

071 

122 

-066 

• 067 

036 

116 

-059 

-ICO 

039 

-055 

-060 

122 

-025 

053 

089 

-194 

-051 

loco 













78 

-022 

-026 

-362 

085 

-029 

-113 

-037 

-066 

-041 

-073 

-002 

- O - i0 

CC5 

-037 

076 

-022 

-032 

-124 

-017 

102 

-OGl 

- C53 

-054 

-026 

- n6 

-022 

115 -083 

080 

054 

-061 

- 04c 

1COO 












79 

-139 

037 

056 

-373 

129 

110 

140 - OC9 

018 

-059 

092 

-053 

044 

-046 

007 

059 

-199 

039 

136 

114 

035 

081 

-022 

-075 

-053 

-139 

022 

053 

112 

022 

-13 e 

093 

055 1000 











80 

085 

128 

032 

195 

-154 

-067 

-007 

026 

-051 

-018 -070 

-053 

029 

-007 

- 0C7 

-118 

121 

-004 

-020 

083 

-155 

160 

-010 

039 

062 

-017 

028 

041 

-053 

195 

052 

-104 

-017 

-274 

1000 










01 

036 

110 

~© 6G 

131 

078 

-423 

012 

-205 

046 

-096 

-102 

-053 

-115 

-062 

-040 

-001 

-032 

-039 

037 

017 

-096 

-114 

124 

008 

009 

-081 

158 

-024 

108 

-124 -078 

059 

154 

-067 

125 

1000 









82 

-071 

075 

215 

C58 

084 

-113 

-570 

042 

-261 

-136 

-023 

004 

126 

107 

018 

-071 

-005 

041 

080 

-214 

-044 

167 

033 

-043 

007 

064 

C14 

-036 

-126 

152 

-047 

-402 

-071 

-068 

103 

050 1000 









83 

237 

-033 

-141 

036 

-037 

010 

-046 

-038 

-054 

-095 

189 

-062 

035 

-048 

-081 

’37 

-041 

-004 

-137 

051 

070 

-120 

051 

-033 

-138 

237 

064 

165 

-001 

-051 

141 

-062 

237 

- 1C4 ' 

-075 

-014 -128 1000 








84 

176 

-043 

-075 

-008 

-046 

128 

072 

-113 

-06 -j 

035 

085 

027 

097 

-061 

155 

-037 

100 

-080 

-221 

201 

- 06 ? 

-067 

0C3 

-043 

097 

-037 

020 

- C65 

-077 

-298 

-013 

133 

-037 

073 

-070 

-135 -?31 

118 

1000 







85 

319 

-026 

-221 

083 

-029 -013 

-037 

-068 

150 -073 

-002 

-048 

112 

-037 

-063 

319 

-032 

-124 

-128 

-034 

036 

-152 

-054 

-026 

-066 

-022 

115 

-083 

080 

-102 

-051 

-048 

319 

- 4)40 

-118 

036 -207 

503 

176 

1000 






85 

-057 

062 

-147 

034 

-075 

012 

-095 

-006 

058 

- C27 

-276 

023 

060 

-095 

077 

-057 

024 

-060 

050 

062 

145 

077 

-072 

062 

-005 

-057 

013 

-064 

-098 

- C62 

036 

023 

090 

-061 

-074 

-007 -088 

-075 

009 

-057 

1000 





87 

271 

-030 

-052 

— o<?8 

-033 

045 

-043 

035 

-048 

-083 

059 

-055 

068 

142 

168 

-026 

-037 

-144 

012 

073 

008 

-090 

-062 

-030 

-019 

-026 

085 

-096 

-054 

-208 

-071 

093 

-026 

097 

-137 

-094 -161 

193 

327 

271 - C56 

1000 




58 

-037 

-043 

-163 

-098 -048 

128 

072 

051 -069 

191 

019 

027 

-168 

072 

-104 

L76 

-052 

-018 

-121 

008 

085 

-005 

008 

-043 

035 

176 

020 

050 

049 

-008 

251 

027 

-037 

015 

-070 

-062 -147 

118 

-061 -037 

-003 

-043 

lOOO 



89 

31S 

-026 

-221 

083 

-029 

-115 

-037 

Ot>3 

-041 

052 

103 

-048 

112 

-037 

-063 

-022 

-032 

-124 

-128 -054 

-031 

-152 

-054 

-026 

-066 

-022 

115 

033 

-023 

-102 

030 -048 

319 

•139 

-017 

036 -207 

503 

176 

659 -057 

271 -037 

1000 


9C 

279 

-058 

001 

-064 

-065 

0C7 

123 

-153 

092 

13 P 

064 

13S 

-074 

020 

-141 

779 

04 r 

110 

-057 

029 

-059 

-199 

029 

-058 

060 

115 

047 

-018 

066 

-029 

-069 -025 

- C30 

022 

-120 

044 -182 

192 

020 

113 

013 

083 

225 

115 

3000 
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TABLES 37 t 38 & 39 


146 


GIVE THE 

PHI COEFFICIENTS FOR THE WRONG ANSWERS 
FOR GROUP B 

LOWER TRIANGULAR MATRIX 


IN THREE PARTS 




1000 . 

327 1000 
ooo ooo 
-090 -097 
-192 -1U 
-226 -155 
-060 11 * 
-056 -105 
02 * -024 
-155 -034 
043 OOO 
-055 OOO 
-060 -225 
-010 -010 
046 031 

OOO OOO 
-031 136 

195 115 

-069 -201 
046 -091 


10 11 12 13 14 15 16 17 16 19 20 21 


26 29 30 31 3 


000 - 
■ - ; 10C0 
OOO 054 1000 
OOC 200 275 1000 

OOO -036 -028 -026 1 COO 

OOO 010 066 130 114 1000 

OOO -045 -095 215 -050 100 1000 

OOO OOP C08 276 066 090 130 1000 

OOO 053 -C 7 -080 -045 -132 -029 -022 1000 

OOO -044 -056 -039 071 -033 099 043 093 1000 

OOO -036 -028 271 -015 -007 122 2C4 -045 212 1000 

OOO 109 157 175 -021 -097 -C71 -004 198 COi -021 1000 

OOO -025 -020 -016 -010 081 -035 -051 228 -049 -010 -015 1000 

oco OOO OOO OOO OOO OOO OOO OOO OOO ooo ooo ooo ooo 

OOC -169 042 -060 -083 -075 117 033 091 -106 -230 091 045 

OOO -023 -04 -077 -044 -OjI -020 094 -067 IOC -044 -063 -C31 

OOO 156 084 555 -030 177 250 276 I'd 074 229 142 -021 

OOO -025 -020 -01.* -010 C31 -035 -051 32 - 49 -010 -015 -OC7 


OOO 

OOO 1000 

OOO -029 1000 

OOO -170 007 1000 

OOO -162 -031 -021 1000 


22 

-080 

114 

OCO 

-036 

-028 

-026 

493 

-007 

-050 

204 

-04 S 

071 -015 

-021 

-CIO 

OOC -083 

145 

229 

-010 

U3 

1000 











23 

022 

130 

OOO 

-oe6 

043 

064 

-06} 

-0113 

242 

160 

-139 -089 

086 

-039 

-044 

OOO 

099 

147 

-120 

-044 

-030 

-063 

1000 










24 

-010 

-0 <7 

oo 

-051 

-040 -037 

-021 

077 

-071 

094 

-065 

102 

-021 

-030 -015 

OOO 

-223 

072 

32 7 

494 

161 

340 

017 

IOOO 









25 

-001 

155 

ooo 

-129 

-002 

020 

056 

-005 

191 

-052 

-045 

042 

056 

-110 -055 

OCO 

027 

-082 

-C?0 -055 

027 

056 

102 

-11C 

1000 








26 

-0?6 

046 

ooo 

063 

226 

C79 

-053 

-0o5 

-040 

-533 

101 

026 

097 

133 

177 

OOO 

074 

-059 

039 -041 

043 

-058 

-112 -033 

019 

1000 







27 

-159 -091 

ooo 

-025 

370 

401 

-010 

-091 

-035 

144 

-034 

-049 

-010 

-015 

-007 

OOO 

045 

-031 -021 

-007 

-037 

-010 

166 

-01 r 

-055 -041 1000 






?• 

-163 

039 

ooo 

033 

-013 

0.7 

049 

-0 6 

079 

c... 

033 

loO 

-0.2 

-023 

187 

OOO 

051 

-097 

-OJJ 

-057 

134 

-Oo2 

037 

-110 

087 

—04 4 

127 

1000 





29 

166 

034 

ooo 

004 

-071 

-06C 

-037 

-069 

092 

-06 1> 

oil 

0('3 

-037 

-C53 

-086 

OOO 

100 

-112 

033 

-026 

-064 

-on 

030 -053 

-Oj4 

-149 

-02C 

-041 

1<XX> 




30 

-159 

-091 

ooo 

-025 

-020 

-Cl j 

-010 

081 

-035 

-051 

22 J 

-04? 

-CIO -015 -007 

oco 

-162 -031 

-081 

-OC7 

196 

-010 

-044 

-015 

-055 

-041 

-007 

-057 

-026 

1000 



31 

—360 

02<- 

000 -057 

138 

-041 

-C24 

105 

-0.0 

06x 

-073 

74? 

-O.l 

-033 

-017 

OOO 

102 

-070 

-048 

-017 

-C34 

-024 

-005 

198 

C47 

006 

-C17 

121 

-060 -017 

1000 


32 

-022 

-304 

coo 

C6l 

019 

034 

-04 4 

-021 

-0J5 

043 

-136 

-049 

-044 

-063 -031 

OOO 

-029 

-060 -090 

236 

-034 

-044 

-020 

r.72 

-132 

-059 

236 

COI 

049 

-031 -CIO 1000 

33 

-053 

-066 

oco 

039 

-04 7 

146 

-064 

040 

162 

.47 

070 

-016 

083 

014 

-04 S 

ooo 

107 


370 

-045 

010 

Oc 3 

030 

116 

C51 

061 

-045 

-089 

-038 -045 

-008 

-13 6 

34 

-139 

-091 

coo 

-025 

-020 

401 

-010 

071 

206 

144 

-032 

-049 

-010 

-015 

-007 

OOO 

045 

-031 

344 

-007 

196 

-010 

-044 

-015 -055 

-041 

-C07 

127 

-026 

-007 

-017 

-031 

35 

C72 

028 

ooo 

-005 

-318 

-139 

090 

-C93 

090 -007 

-090 

178 

090 -052 

063 

OOO 

049 

174 

-075 

-115 

-091 

090 

Oil 

-052 

107 

-107 

-115 

140 

-093 

-115 

-Olo 

-015 

36 

028 

-081 

ooo 

-069 

063 

-100 

100 

-034 

074 

020 

-005 

-007 

-057 

031 -040 

ooo 

020 

006 

125 

-040 

155 

100 

081 

255 

-052 

014 

-040 -032 

-145 

-040 

009 

-053 

37 

034 

003 

ooo 

-049 

017 

-043 

-071 

-051 

044 

033 

035 

-024 

-071 

-101 

147 

ooo 

065 

101 

-073 

-050 

ol9 

-071 

067 

-101 

06.' 

017 

147 

-033 

-003 

147 

-024 

-107 

38 

-039 

041 

ooo 

-044 

-034 

-032 -018 

-153 

060 

025 

-056 

145 

-018 

-026 -013 

coo 

079 -054 

-037 

-013 

071 

-018 

-077 

-026 

-095 

055 

-013 

114 

-046 

-013 

-029 

-054 

39 

-185 

-.40 

ooo 

061 

123 

256 -044 

160 

-149 

146 

-067 

-102 

-044 

072 

-031 

000 -084 

-131 

104 

-Ojl 

028 

-044 

035 

072 

-032 

057 

236 

098 

-031 

-03 1 

051 

15 ’ 

40 

-081 

028 

ooo 

229 

-045 

-041 

-024 

020 

140 

-115 

045 

159 -024 

-033 

-017 

ooo 

-086 

051 

-048 

-017 

021 -024 

-100 

~0?3 

047 

105 

-017 

-13C 

073 

-017 

-038 -070 

41 

oy 2 

-759 

ooo 

-020 

062 

-064 

-080 

C38 

-049 

-070 

-248 

-036 

183 

-114 

-057 

ooo 

-262 

-092 

038 

129 

-073 

-060 

-0J2 

167 

-0C6 

184 

-057 

-207 

019 

-057 

-C44 

154 

42 

CT3 

093 ’* 

■ ooo 

005 

-103 

065 

037 

C63 

-004 

076 

-140 

-424 

-090 

-123 -063 

ooo 

024 

-080 

075 

115 

00, 

037 

026 

052 

027 

-C09 

-9o3 

-107 

-014 

-063 

-144 

-032 

43 

-005 

-093 

ooo 

-003 

-075 

-069 

-039 -044 

149 

147 

-045 

-012 

-039 

-056 

-028 

090 

-131 

116 

026 

-028 

-073 

166 

016 

092 

-097 

035 

-028 

051 

-on 

-028 

-06? 

-039 

44 

021 

003 

ooo 

069 -081 

040 

-042 

050 

137 

058 

Oil -C92 

-012 

-060 

-030 

coo 

126 

-C54 

013 

-OJO 

-085 

-042 

-009 

••OtO 

0o5 

010 -030 

016 

057 

-030 

-O 60 

-05 4 

45 

-056 

059 

ooo 

063 -049 -045 

-026 

-012 

-033 

-127 

023 

-039 -026 

-037 

-018 

coo -oco 

-077 

-053 

-01. 

101 

-do 

-110 

••037 

020 

079 

-01, 

0.7 

-066 

-01 j 

149 

-077 


> 

bd 


o 


1000 

162 

024 

117 

137 

162- 

029 

-102 

-043 

160 

071 

042 


1000 

-115 1000 
-040 -123 1000 
-050 084 -05 > 1000 

-C13 111 059 027 1000 

-031 -157 -112 049 -054 1000 

-017 003 009 005 -029 -070 1000 

-0,7 -009 057 121 009 O’.-O 040 1000 

115 013 0,3 14 -lil -0 0 -C-63 069 1000 

-O '. 054 106 -074 122 -03 1 070 0>6 030 1000 

745 016 -043 CIO 10. r 19 1.1 022 033 366 10-30 

-01. -0-.5 -03 -C*3 —03 7 C34 IV.' 013 -010 -059 C40 ldO 


•-then* nvuibers should b« multlpllad ay 10~~ 



1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

n 

12 

13 

14 

15 

16 

17 

18 

19 

1 20 

?! 

22 

23 

1 24 

25 

26 27 

28 

1 29 30 31 

32 33 3 ^ 35 36 37 36 39 

1 4C 

41 

42 

43 

44 

45 

' 46 

-039'* -058 

000 

140 -034 

212 

398 

141 

080 

025 

-056 

0:0 

-018 

-026 

-013 

000 

-162 

101 

176 

1 -013 

207 

-018 

-cn 

' -02C 

-095 

-072 -013 

114 

-046 -013 

i -0?s 

-054 16.? 573 07 059 -087 -02? 

' -054 

-029 

009 097 

122 

108 -032 

47 

115 

-084 

000 

— 063 

-049 

-045 

217 

-012 

-088 

-127 

-030 -039 

-026 

-037 

-018 

000 

-405 

-077 

099 

401 

295 

-025 

-0?3 

:e 7 -058 

-103 -013 

010 

060 

1 -018 -041 

C34 -026 -018 -C65 084 039 -032 

145 

-041 

162 

! 139 

-069 

-074 

-045 

*8 

-045 

-014 

000 

-164 

010 

-198 

053 

-054 

on 

-147 

043 

-027 

053 

-043 

-196 

000 

106 

-032 

-?31 

037 

-135 

-n .3 

-Ot7 

-043 

104 

-043 037 

-006 

134 

037 

084 

096 -058 -156 -116 051 -065 

065 

-023 

-127 

U6 

075 

-063 -C39 

-101 

49 

-107 

-008 

000 

001 

148 

-003 

-057 

-159 -033 

063 

no 

-007 

-057 

031 

-040 

000 

202 

-053 

-117 

-040 

001 

-057 

173 

-081 -010 

062 182 

099 -Ol 2 

-eto 

' no 

123 071 -040 -C83 023 158 

053 

065 

-091 

016 

-074 

042 

017 

-ooe 

50 

-034 

020 

000 

054 

-053 

113 

248 

020 

-001 

163 

-087 

-056 

248 

-040 

-020 

000 

-no -034 

084 

-020 

170 

248 

-•J.9 -040 

216 

058 -020 -084 

-071 

-020 

-045 

019 038 -020 -036 -023 -135 

-034 

019 

132 

134 

105 

-075 

025 

275 

51 

004 

006 

000 -033 

-090 

-14 e 

-0fl4 

-C87 

-023 

-061 

021 -003 

045 

-028 

-059 

000 

067 

086 

-040 -059 

-303 

045 

021 

054 

-070 

-217 -059 -C01 

114 

-059 

-052 

-252 -142 -059 O64 -086 -015 

-104 

038 

113 

010 

-064 

199 

204 

-148 

52 

-142 

-018 

000 

194 

046 

060 

-037 

034 

-054 

051 

041 

003 

-037 

-053 

-026 

000 -037 

-n2 

144 

-026 

006 

-037 

-096 

100 

030 

049 -026 

070 

-004 

-026 

078 

049 -038 -026 122 -078 116 

130 

129 

215 

131 

040 

-100 

057 

060 

| 53 

-137 

-136 

000 

-090 -042 

-026 

230 

040 

-017 

007 

070 

066 

-064 

014 

-045 

000 

022 

029 

095 

-045 

106 

033 

-272 

•091 

-027 

-029 -045 

-051 

-03 e 

162 

086 

-0&1 -022 162 -013 071 -025 

C42 

-191 

086 

-08 1 

-024 

-050 

-071 

060 

54 

-C34 

-047 

coo 

054 

248 

113 

-028 

020 

-001 

088 

-087 

021 

-028 

-040 

-020 

000 

122 

019 

-057 

-020 

-101 

-028 

143 

-040 

-075 

-027 370 

058 

046 

-020 

132 

226 038 -020 034 -109 093 

-034 

123 

132 

-009 

-103 

-075 

-031 

-019 

55 

140 

-105 

000 -055 

-101 

-093 

-053 

-025 

-on 

-033 

-043 

073 

-053 

-075 

196 

000 

-050 

023 

C61 

196 

081 

-053 

-030 

043 

-278 

-058 -037 

049 

147 

-037 

-034 

213 -134 -037 -.’31 -102 065 -065 

151 

021 

013 

-0J4 

-C05 

-008 

004 

56 

091 

-010 

000 

-057 

009 

-049 

063 

014 

-no 

037 

-oce 

O.H 

-073 

-105 

■052 

000 

002 

190 

-150 

141 

C49 

-073 

0C9 

090 

-134 

-29? 141 

-019 

-071 

-052 

-030 

036 C38 -052 - r 54 Oil 023 

022 

-015 

-030 -009 

-1C7 

029 

C51 

031 

57 

043 

-038 

000 

C53 

-007 

028 

-045 

221 

095 

-022 

-006 

1:5 

139 

-045 

-032 

000 

-177 

-067 

190 -032 

198 -0*5 

-139 

-065 

-045 

-068 -032 

-062 

041 

228 

C45 

073 -091 -032 -044 -062 086 

-056 

002 

163 -008 

-140 

-121 

-131 

028 

58 

117 

-044 

000 

001 

101 

-087 

035 

077 

-053 

-001 

039 

-221 

-091 

-040 

-064 

000 

105 

151 

-122 

113 

-122 

161 

-093 

049 

-083 

022 -064 

-507 

-01? 

113 

-065 

057 041 -064 -134 -002 -024 

-009 

-132 

096 

-040 

165 

-037 

-021 

-03-7 

59 

oes 

C24 

coo 

-023 

-084 

-077 

-044 

-157 

-085 

-0C9 

071 

213 

145 

-C63 

-031 

000 

-029 

-OtO 

007 

-031 

028 

-044 

n 35 

072 

018 

-001 -031 

098 

-112 

-031 

-070 

-060 -081 -031 CJ2 -053 -055 

101 

061 

-070 

006 

-060 

1 16 

C19 

034 

60 

-076 

-112 

000 

144 

123 

145 

145 

070 

044 

-009 

071 

056 

145 

072 

-031 

000 -034 

-060 

298 -031 

090 -044 

-076 -063 

-032 

-059 -031 

001 

049 

-031 

051 

-060 -081 -031 -042 123 -159 

-054 

cei 

051 

-043 

no 

033 

-054 

054 

61 

-360 

-128 

COO -057 

132 

149 

-024 

-050 

-080 

150 

-073 

-021 

-C24 

-033 

-017 

000 

008 

-070 

-049 

-017 

-084 

-024 

090 -033 

-124 

-093 441 

121 

-060 

-017 

-038 

051 -008 -017 -C >5 009 -024 

-029 

172 

-033 

-128 

Cl8 

070 

057 

-041 

62 

-113 

-128 

000 

176 

097 

113 

-028 

-047 

092 

083 

215 

098 

248 

354 

-020 

000 

042 

-084 

084 

-020 

-010 

-028 

043 

-040 -075 

058 -020 

058 

-071 

-020 

-045 

-084 n8 -020 034 063 -135 

-034 

123 

132 -009 

-172 

039 

025 

-049 

63 

067 

064 

000 

003 

057 

-121 

012 

-111 

001 

-006 

-028 

126 

012 

018 

-077 

coc 

123 

-100 -037 

-077 

044 

012 

035 

-069 

097 

-100 095 

-010 

031 

-077 

-097 

037 -476 -077 000 057 009 

-035 

082 

059 -028 

-092 

-042 

106 

022 

64 

066 

052 

000 

-486 

-009 

-112 

073 

-030 

-030 

024 

078 

004 

-063 

007 

052 

000 

078 -036 

-270 

052 

040 -063 

071 

-188 

134 

-001 052 

019 

-220 

052 

030 

066 -157 -141 C29 -055 052 

090 

-036 

-233 

-062 

-200 

027 

-104 

-031 

1 65 • 

-27; 

-113 

000 

054 

-053 

275 

-028 

086 

-001 

088 

-037 

021 

-028 

-040 

-020 

000 

-118 

-084 

225 

-020 

080 

-026 

04 3 

157 

-C75 

-027 -020 

058 -071 

-020 

132 

019 n8 370 -312 -023 -135 

-034 

123 

-045 

062 

105 

039 

025 

1X3 

66 

-036 

098 

000 

069 -062 

-127 

-072 

-034 

-151 

-0*52 

-022 

043 

-072 

-004 

-051 

000 

-127 

04 3 

-147 

-051 

-123 

-072 

-104 

-103 

057 

220 -051 

064 

-183 

-051 

150 

043 -073 -051 096 -279 033 

138 

-061 

-027 

037 

0C7 

091 

056 

n6 

67 

058 

032 

000 -034 

-082 

-117 

-149 

003 

-043 

-191 

035 

019 

-025 

052 

-105 

000 

037 

-075 

-115 

-105 

-u90 

-149 

-016 -036 

-002 

014 -105 

004 

094 

-105 

079 

017 -181 -105 015 -001 -716 

020 

-029 

-001 

-105 

-045 

009 

-001 

101 

68 

002 

021 

000 

-035 

-090 

-083 

-047 

-410 

024 

-034 

182 

076 

-047 

190 

-033 

000 

048 

061 

-096 

-033 

-051 

-047 

-041 

-067 

037 

033 -033 

065 

033 

-033 

-075 

061 -204 -033 018 041 -028 

-058 

-140 

C40 

-116 

-063 

-051 

-135 

-083 

69 

-080 

-129 

000 

188 

248 

-026 

-015 

-007 

-050 

-072 

-045 

-069 

-015 

-021 

-010 

000 

-083 -044 

-030 

-010 -053 

-015 

-063 

-021 

056 

097 -010 

-082 

-037 

-010 

-024 

-044 -064 -010 -037 -057 -071 

-018 

-044 

301 

183 

-090 

-039 

152 

271 

i 70 

142 

052 

000 

-111 

010 

-165 

-012 

-035 

-042 

-653 

-016 

-092 

-134 

-018 

077 

000 -052 

-082 

-150 

077 

-034 

-134 

-070 -018 

031 -086 -095 

041 -083 

077 

-059 -082 -160 -095 o;o -029 -009 

035 

-082 

-215 

028 

-030 

-103 

-05 S 

05c 

71 

-114 

-010 

000 

109 

157 

175 

-021 

-010 

052 

094 

-065 

102 

-021 

228 

-015 

000 

-014 

072 

327 

-015 

-075 

340 

017 

228 

-015 

027 -015 

070 

-053 

-015 

198 -063 n8 -015 C38 031 -101 

-026 

072 

•033 

-m -038 

C92 

-060 

-037 

(72 

-005 

106 

000 

271 

039 

053 

-039 

-044 

-062 

-022 

030 

-134 

-039 

092 

-028 

000 

no 

-117 

026 

-028 

-073 -039 

016 

; -056 

067 

-092 -028 

051 

-on -028 

070 

038 -050 -028 -020 -087 -131-046 

ne -063 

-105 

-239 

-104 

047 

053 

; 73 

-013 

-055 

:oo 

091 

-084 

010 

049 

-086 

-010 

-079 

129 

305 

-082 

070 

-057 

000 -025 

-048 

-100 -057 

-079 

-082 

-078 

-023 

018 

-084 127 

054 

-041 

-057 

121 

050 -051 -057 238 -113 039 

007 

-046 

121 • 

-139 

-140 

-217 

-084 

CIO 

! 7 * 

-025 

-014 

000 

112 

-037 

-013 

161 

-074 

032 

067 

034 

164 

035 

-130 

113 

uoo 

032 

010 

007 

-064 

125 

035 

013 

-040 

017 

-055 -064 

-117 

142 

-064 

096 

057 n4 -064 024 -002 -093 

0-34 

010 

015 ■ 

-073 

-024 

-038 

-263 

-057 

, 75 

203 

044 

000 

-155 

-025 

-154 

042 

-096 

012 

046 

-152 

146 

04? 

-078 

-245 

000 -015 

054 

-11J 

030 

-102 

042 

180 

060 

-033 

-129 030 

-117 

025 

030 

-057 

126 -155 -245 C33 104 043 

052 

-164 

-131 

029 

-033 

-047 

-177 

-611 

!76 

065 

-119 

coo 

C45 

256 

064 

-063 

-012 

-in 

-023 

-035 

-047 

-063 

335 

-044 

000 

013 

-076 

024 

-044 

ns 

-063 

-048 

-089 

062 

161 -044 

075 

-159 -044 

-005 

035 -056 -044 on 034 026 

-077 

202 

-100 

007 

026 

-166 

-ISO 

064 

77 

-14 1 

-155 

coo 

200 -049 

-045 

-026 

-012 

013 

035 

-090 

043 

-026 

-037 

-018 

COO -405 

034 

-053 

-018 

004 

-026 

06V 

-037 

099 -012 -018 

010 

-066 

-Clfi — C41 

034 -026 -Cl 9 0 4 -100 039 

-032 

-077 

149 

245 

-010 

296 

040 

129 

' 78 

-080 -007 

000 

168 -025 

-026 

-015 

-007 

-050 

-072 

139 

-069 

-015 

-021 

705 

COO -083 

-044 

-030 -010 

113 

-015 

-06} 

-021 

056 

252 -010 

049 

-037 

-010 

-024 

-044 -064 -0.10 090 -057 069 -018 

-044 

301 

052 

-090 -039 

152 

271 

79 

137 

152 

000 -051 

-142 

-121 

134 

-023 

-082 

-039 

060 -009 

-no 

018 

095 

000 

052 

123 

-224 

-077 

-155 

012 

03 V 

-069 

-0S5 

-100 -077 

-OlO 

033 -077 

-019 • 

-100 -068 -077 051 -009 -024 

-035 

-054 

059 -091 

060 

058 

059 

022 

so 

-050 

-078 

000 

C84 

Oil 

-176 

-100 

-018 

037 

-155 

049 

-027 

023 

033 

103 

000 

078 -023 

-142 

-071 -037 

-100 

045 

-C55 

025 

017 -071 

018 

059 

103 

-002 

-023 030 -071 -034 -006 027 

-022 

-023 

-002 

031 

004 

087 

043 

-176 

, 81 

-095 

-136 

000 

039 

-042 

-026 

-064 

005 

033 

-153 

-037 

-097 

-064 

014 

-045 

000 

022 

-026 

-056 

-045 

-230 

-064 

-099 

-091 

051 -029 162 

-089 -038 -045 

-008 

303 148-045-049 -020 -105 

042 

-026 -008 -043 

049 -050 -015 

ns 

I® 2 

318 

162 

000 

-156 

-162 

-381 

-108 

-110 

-037 

-266 

-010 

-060 

-108 

-039 

033 

000 

094 

101 -471 

038 

-124 

-271 

087 

-270 

029 

-033 -191 

007 

O69 

-191 

-121 • 

-020 -094 -191 -CC7 -04? -142 

-067 

-202 • 

-121 - 

-127 

-034 

078 ■ 

-031 -000 

83 

-015 

-077 

000 

156 

225 

251 

-030 

no 

-102 

135 

-093 

-070 

-030 

142 

-021 

000 

-245 

007 

469 

344 

146 

229 

-123 

512 

-090 

-040 -021 -033 

-077 

-021 -048 

007 020 -021 -075 044 -002 

-037 

104 • 

-048 

105 

140 

026 

-087 

-053 

84 

-164 

-077 

000 

270 

084 

403 

-030 

-014 

C74 

276 

001 

002 

229 

142 

-021 

000 

-020 

-090 

337 

-021 

061 

-030 

175 

-043 

-158 

-CKO -021 

-033 

033 

-021 

-048 

-090 095 344 -140 -036 140 

-037 

104 ■ 

-048 -029 

010 • 

-080 

212 

-053 

: 85 

-226 

-155 

000 

331 

275 

652 

-026 

-012 

013 

197 

023 

-0.19 

-026 

307 

-018 

000 

026 

-077 

4o3 

-018 

198 

-026 

-023 

-037 

-136 

079 401 

163 

060 

-018 

-041 

145 146 401 -213 -006 039 

-032 

256 ■ 

-C41 -064 

065 -069 

040 

-045 

j 86 

-039 

-045 

000 

012 

-103 

-095 

103 

-084 

037 

-039 

010 

015 

108 

-077 

-038 

000 

-047 

-040 

-026 -038 

018 

103 

055 

039 

100 

-215 -033 -049 -069 -03 8 

121 • 

-040 047 -038 C47 -059 007 

067 

-101 • 

-037 -042 

034 • 

-on 

-031 -095 

87 

002 

-045 

000 

356 

073 

375 

-032 

073 

058 

no 

-010 -033 

-032 

129 

-023 

000 

-074 

-004 

3.11 

323 

045 

214 

080 

304 

-039 -052 -023 

012 

023 

-023 

-051 

085 074 -C23 -049 028 -019 -039 

088 ■ 

-051 -047 

111 

ns 

096 

-056 

! 38 

030 

010 

000 

111 -063 

-063 

-036 

-097 

-045 

-114 

-029 

018 

-036 

109 

-025 

900 -039 

-023 

-073 

-025 

-055 

-036 

045 

109 

107 

132 -025 ■ 

-198 

099 

-025 

086 ■ 

-023 -0?5 -025 -005 001 -049 

-044 

-023 

066 

212 

-051 -003 

069 

068 

89 

-217 

-010 

000 -051 

157 

175 

-021 

077 

174 

-004 

-065 

001 

-021 

-030 

-015 

ooo 

091 

-063 

142 -015 

043 

-021 

-083 

-030 

-no 

027 -015 

164 

-053 

-015 

-033 ■ 

-O63 118 4,94 -143 -081 -001 

270 

072 • 

-033 

073 

-036 -056 

217 

-037 

90 

104 

-069 

coo 

004 

046 

-066 

-037 

-121 

092 

-125 

041 

-057 

-037 

253 

-026 

900 

-024 

-112 

-077 

277 

006 

-037 

030 

IOC- 

036 

-017 -026 -097 

270 

-026 

-060 

045 -100 -026 -040 -078 -003 

-046 

-112 

078 

075 

-068 • 

-100 

057 

060 


i_3 

> 

bd 

\ r * 


CO 


• those nuabors should be aiultiplled by 10“- 


CO 


45 n 48 49 50 51 52 *53 54 55 55 57 50 55 60 61 62 63 64 65 66 67 60 69 70 71 72 73 74 75 76 77 7- 7° 80 81 02 83 65 ,6 .7 -3 09 90 

45 1000* 

47 23.2 1000 

48 -207 093 1000 

49 -070 -100 102 1000 

50 -034 -049 030 -109 1000 


51 

-104 

-073 

007 

-OOC 

-090 1000 









52 

-046 

060 -006 

055 

163 

005 

1000 








53 

161 

-026 

-053 -020 

193 

046 

-038 1000 







54 

-034 

-049 

010 

143 

097 

-090 

163 

038 

1000 






55 

-065 

101 

-0-31 

052 

-101 

035 

-134 

058 

-010 

1000 





56 

Oj2 

112 

-139 

013 

-140 

C33 

-137 

-121 

034 

003 

1000 




37 

-056 

023 

043 

-176 

114 

-073 

-037 

070 

014 

258 

-028 

1000 



36 

-009 -087 

-042 

037 

-037 

-010 

-072 

114 

-937 

-081 

054 

-053 

1000 


59 

-054 

034 

096 

123 

019 

033 

-031 -026 

019 

090 

-066 

-067 

-226 

1000 

60 

256 

034 

-028 

006 

-084 

-010 

-031 

034 

-034 

028 

-066 

071 -035 

081 1000 

61 

-029 

-041 

034 

110 

-045 

113 

8 

a> 

132 

021 

145 

-073 

015 

-970 -070 1000 


62 

-034 

-0< 5 

010 

-109 

-053 

121 

-07 1 

-122 

097 

-010 

-065 

-087 

-106 

123 

123 

-045 

1000 




























63 

-135 

C22 

194 

-009 

-010 

180 

-124 

018 

-CIO 

005 

-053 

060 

-014 

128 

128 

059 -010 

1000 



























64 

-134 

049 

139 

157 

-009 

-137 

-103 

002 

-084 

-005 

145 

028 

014 

-036 

-133 

-058 -233 

-045 

1000 


























65 

192 

-049 

-OSO 

-023 

097 

051 

046 

038 

-053 

-OlO 

009 

0 14 

-037 

019 

019 

132 -053 

-709 

-233 1C00 

























66 

C?5 

035 

-012 

-022 

013 

-2*02 

-005 

-073 

-062 

-123 

-099 

C79 

033 

094 

-112 

-115 

-137 

027 

136 

013 

lOOO 
























67 

-032 

-117 

131 

-040 

121 

-002 

-064 

-002 

-082 

-050 

-042 

-055 

-002 

no 

064 

079 -082 

051 

008 -014 

045 

1000 























68 

-058 

-093 -008 

153 

-090 

093 

033 

005 

-090 

065 

-186 

-145 

021 

061 -073 

040 

107 

125 

040 -090 -034 

052 

1000 






















69 

-018 -026 

-113 

-057 

248 

045 

173 

033 

-026 

-053 

-073 

-045 

C'T’ 

-044 

-044 

-024 

-028 

012 

-O63 -028 

066 

-025 

-047 

1900 





















70 

033 

193 

124 

009 

-123 

038 

-031 

-124 

-190 

errb 

-045 

028 

-107 

054 

054 

-059 

-123 

025 

111 -057 

039 

098 

090 

-‘312 

1C00 




















71 

-026 -037 

-161 

-081 

-040 

064 

-053 

-091 

-040 

-075 

-105 

067 

-040 

-063 

2C7 

-033 

157 

-069 -090 

-040 

-004 

-036 -067 

-321 

-C1D 

1000 



















72 

-048 

-069 

073 

042 

039 -013 

165 

-170 

264 

-140 

029 

030 

-140 

038 

038 

070 

152 

053 

-197 

-075 

-022 

009 

-051 

-039 

092 

092 

1000 


















73 

0C7 

010 

079 

-0J2 

-004 

-034 

014 

-013 

-013 

-037 

052 

Obi 

-102 

050 

001 

037 

130 

021 

-017 

-004 

099 

004 

ill 

049 

073 

-023 

051 1000 

















74 

094 

-013 

-001 

037 

101 

-106 

142 

005 

032 

-081 

-043 

084 

059 

010 

057 

-065 

-037 

-014 

U6 

0J2 

-036 

059 

111 

035 

-016 

-130 

-037 

079 1000 
















75 

-267 

074 

230 

043 

-131 

-005 

-057 

-093 

081 

C25 

1C6 

060 

021 

054 

-164 

-057 

-025 

001 

157 

-131 

-005 

-047 

135 

-152 

012 

-070 

-126 

134 

021 

1000 















75 

-077 

-023 

-012 

-053 

124 

-131 

030 

073 

124 

-079 

-111 

023 

-019 

-076 

C35 

-100 

124 

142 

-009 

-038 

018 -069 

Oil 

086 

073 

017 

C16 

-001 

055 

0C9 1000 














77 

212 

-045 

-101 

-003 

lU 

003 

060 

060 

-049 

004 

-049 

-COO -087 

-077 

034 

149 -049 

022 

-112 

113 

197 

028 

C2J 

271 

-022 

-037 

-069 

010 -013 

-154 

064 

1000 













78 

-018 

-026 

-278 

-C57 

248 -084 

173 

083 

-022 

113 

-073 

-0*5 

-091 

-044 

-044 

-024 -028 

-110 -063 

-023 

066 

••025 

-047 

493 

-012 

-C21 

-039 

04° 

035 

-347 

-063 

271 

1000 












79 

065 

022 

-044 

-425 

-010 

086 

-072 

-063 

-076 

-155 

177 

-2C6 

016 

-054 

-145 

-019 

-o?.o 

034 

086 

-076 

027 

110 

039 

-110 -034 

-069 

-042 

147 

077 

-012 

-073 

022 

012 

1000 











60 

-022 

041 

073 

224 

-191 

149 

007 

-078 

Oil 

043 

033 

-040 

053 

069 

023 

-002 

Oil 

-005 

-066 

-l:.4 

013 

070 

-013 

-100 

123 

033 

-014 

-174 

022 

004 

-136 -031 

023 

-212 

1000 










81 

042 

-112 

038 -065 

038 

-367 

024 

-022 

118 

-038 

078 

-037 

114 

-136 -026 

006 

-042 

-123 

-038 

038 

247 

070 

-040 

•983 

052 -091 

071 

139 

041 

-042 

-4)13 

146 

033 

089 -042 

1000 









82 

-067 

COO 

071 

0C8 

-251 

021 

-690 

-141 

-162 

142 

183 

-069 

054 

040 

-020 

-121 

015 

091 

212 

-162 

133 

192 

-OOO 

-IOC 

143 

-155 

-122 

-035 

-067 

094 

040 

-090 

-ior. 

130 

092 

-OOO 

10o0 








83 

-037 

251 

-146 -036 

-057 

-040 

254 

-131 

-057 

-023 

-010 

-093 

071 

007 

104 

-048 

084 

-162 

-130 

084 

-006 

-178 

-004 

-330 -025 

327 

026 -033 

-122 

-013 

C24 

-053 -030 -099 

-079 

-131 

-305 

1000 







84 

176 

-053 

-146 -036 

-057 

-040 

144 

020 

-057 

-023 

-150 

-093 

-058 

201 

2C1 

-043 

225 

025 

-200 

225 

-077 

-305 

-0C4 

-030 

-150 

142 

132 

-033 

-058 

-113 

024 

-053 

-030 -037 

-079 

-131 

-305 

204 

loco 






65 

212 

-045 

-198 

176 

-049 

-148 

060 

060 

275 

-093 

-049 

-OOO 

-161 

-077 

145 

149 

113 

-050 

-192 

275 

-127 

-169 

073 

-026 

-236 

175 

053 -067 

-013 

-154 

152 

-045 

-026 

-193 

-031 -02 6 

-286 

251 

403 

1000 





86 

067 

095 

089 

093 

074 

-021 

COO 

094 

-013 

Old 

-271 

CIO 

-054 

202 

142 

-087 

074 

-052 

007 

074 

-000 -073 

-OOO 

-054 

130 

039 

055 

077 

148 

094 

-040 

0?5 -054 

-091 

106 

094 

043 

-028 

139 

-095 

1000 




37 

-CJ9 

089 

-125 

020 

-061 

067 

127 -068 

073 

-035 

-094 

-099 

106 

-096 

088 

-051 

073 

-062 

-171 

073 

-073 

-205 

-102 

-032 -053 

304 

116 

-114 

-077 

-002 

030 -055 

-032 

-180 -039 

-139 

-197 

437 

311 

375 

-039 

1000 



88 

-044 -063 -018 

001 

176 

138 

099 

039 

054 

-055 

-057 

-029 

—222 

-023 

-107 

-057 

054 

-051 

-184 

176 

003 

076 

124 

138 

104 

-051 

180 

149 

056 

017 

111 

200 

1 6 

003 

-025 

039 

-084 

-073 

-073 

06 i 

156 -078 

1000 


89 

270 -037 

-043 -031 

-C40 

064 

100 

014 

-040 

-075 

-105 

-065 

-040 

-063 

-063 

198 -040 

018 

-OfO 

157 

-004 

-124 -067 

340 

069 -030 

092 

-116 

049 

-21? 

017 

-037 

-021 -069 

-055 

014 

-155 -043 

142 

175 

-077 

-046 -051 

1000 

90 

-046 

060 

064 

-012 

046 

114 

087 

-033 

046 

006 

-129 

-116 -072 

-112 

-112 

-060 

163 

C31 

-C45 

-071 

-125 

-C64 

262 

;.93 

124 

100 

077 

070 

035 

025 

156 

106 

178 

-020 

007 

024 

-OOO 

033 

033 

C60 

069 

023 

479 

100 1000 
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TABLE 39 
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TABLli 40 
I TM ANALYSIS 





Selection Ratios 


Mean 

Scores 


c 

n 

Item 

r b 

D* 

D 1 

D 2 

14 

S) 

M* 

Mi 

M 2 

M 

3 

c r 

12 

.345 

.35 

.20 

.35 

.10 

13.5 

11.4 

11.4 

12.1 


20 

. 269 

.53 

. 04 

.04 

.39 

12.8 

9-2 

10.0 

11.9 


26 

.140 

o38 

.15 

.34 

.13 

12.6 

12.1 

11.9 

11.9 

c. 

1 

.384 

.09 

.83 

.04 

.02 

14.2 

12.1 

13.4 

10.0 

1 

2 

.456 

.25 

.61 

.09 

.04 

13.9 

12.0 

9*3 

12.0 


8 

.419 

.29 

.58 

.01 

ol2 

13.6 

11.5 

9.0 

12.4 


28 

.378 

.38 

.25 

.32 

.06 

13 = 3 

11.5 

12.0 

9.5 

c. 

7 

. 336 

.14 

.04 

.31 

.51 

13.7 

10.8 

11.6 

12.2 

5 

22 

.001 

.07 

.02 

.06 

.84 

12.2 

8.3 

11.2 

12.4 


23 

.165 

.58 

.20 

.17 

.04 

12.5 

11.8 

12.4 

8.8 

Of 

9 

.135 

.70 

.11 

.17 

.03 

12.4 

11.4 

12.3 

10.0 

b 

27 

.388 

.85 

.03 

.09 

.03 

12.5 

10.0 

11.1 

8.7 

0-i n 

21 

. 40 4 

.32 

.07 

.37 

.23 

13.5 

11.2 

11 . 4 

11.9 

10 

25 

.329 

• 55 

.24 

.19 

.02 

12.9 

11.5 

11.5 

9.0 


24 

.377 

.86 

.01 

.07 

.06 

12.5 

8.0 

10.7 

10.7 

c x. 

3 

17 

o304 
. 59-6 

0 

00 VJ\ 

r— 1 O 

O OO 

.23 

.03 

.41 

.09 

13.1 

15.2 

10.0 

12.0 

11.2 

12.2 

12.0 

10.9 


30 

<>158 

»74 

.01 

.15 

.10 

12.4 

13.0 

! — 1 
CO 

11.0 

c k 

5 

.295 

.22 

.05 

.61 

.12 

13.4 

12.9 

11.8 

11.5 

Hr 

14 

.405 

.46 

.02 

.10 

.40 

13.2 

12.0 

12.4 

11.1 


19 

.463 

.31 

.09 

.14 

.47 

13.7 

10.8 

11.5 

11.6 

c„ 

10 

.320 

.17 

.22 

.06 

• 55 

13.6 

C\2 

0 
1 — 1 
1 — 1 

12.4 

12.1 

i 

11 

. 334 

.48 

.20 

.30 

.02 

13.0 

11.8 

11.4 

9.7 


16 

„447 

.81 

.01 

.02 

.14 

12.6 

8.0 

9o7 

10.7 


29 

.275 

.83 

.08 

.06 

.02 

12.4 

11.1 

11.1 

8.7 

S 

4 

. 425 

.17 

.10 

.01 

.72 

14.0 
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APPENDIX. B 

THE ADVANCE CLASSIFICATION OF THE ALTKENAT I'VE S 
IN THE EXPERIMENTAL TEST 

The discussion which fellows presents a detailed i tem-by-item 
account of the procedure used in the construction of this experimental 
testo The format of this discussion involves: 

lo Giving the reading selection as it Is required© 

2. Giving each item in its entirety in the format it was given 
to the examinees; except that in this discussion the cat- 
egories of the items and the foils are indicated for the 
convenience of the reader. 

3c The reason for classifying the item by Bloom’s Taxonom y , 
and the foils by the Guidelines as indicated, are given 
following each item. 

4. Items 19 to 24 inclusive form a special case and will, 
therefore, be dealt with as a unit. 

5. In the classification of foils no one item contained, by 
arbitrary practice, two foils from the same category. 


THE EXPERIMENTAL TEST 


Directions for Examinees: 


Answer all questions in Part- I on the basis of the reading 
selections given. 

First Reading Selection 


Source : Dexter, Lewis Anthony; The Tyranny of Schooling , 

N.Y., Basic Books, 1964, p. 1. 

Most people in our society at one time or another suffer 
humiliation, shame, or at least severe apprehension because of 
one great fear: they are afraid that other people may think 

that they are stupid. This fear of being regarded as stupid 
frequently underlies inferiority complexes, self-contempt, 


self-depfeciut i on , and despair. 

Our society teaches contempt for stupidity and fear of being* 
regarded as stupid through one centra], institution and its 
auxiliaries. This institution Is compulsory schooling. It is 
aided by such auxiliary practices as compulsory written 
examinations for admission to many jobs, intelligence testing, 
and the like. 


1. From the above article we may conclude that if society does 
not reduce its contempt for stupidity: 

(Bloom’s Category 4.20) 

A. Emotional problems will continue to be on the increase. 
(OG) 

*B. The development of creativity will continue to be 
restricted . 


Co Mutual co-operation will continue to be difficult to 
obtain. (lA) 

1). Economic power will continue to be confined to a 
minority group. (irr) 

This item was classified as an analysis (4. 20 ) item on the 
grounds that it requires the examinee to display ’’skills in comprehend- 


ing the interrelationship among ideas.” (Bloom: p. 206). The examinee 
is expected tc realize that, contrary to popular myth, creative people 
do not display ’’inferiority complexes, self-contempt, self-depreciation 
and despair” to tne same extent as is found in the population at large. 
For this reason, the development of creativity and the development of 
contempt for stupidity would be expected to be inversely related. 

Since, in the stem, the variable ’’contempt for stupidity” does not 
change, it follows logically that the status of any related variable 


should show no change, 
he should have been able 
Foil 1A (or ll) n , 


If the examinee did net know this relationshi 
to arrive at it from the logic of the foils, 
on the basis of the symbolism used in 


P 


? 



an increase in one variable without a correspond in, 


r 

•o 



not validly warrant the conclusion that contempt for stupidity will 
increase. It is adding incorrect information to the answer to suggest 
a change in one variable without an explicit statement concerning change 
in the appropriate direction in the other variable. Hence the foil is 
classified as an Over Generalization (00). 


ness and contempt for stupidity. Unlike the case for creativity, there 
is no valid reason to assume such a relationship for co-operativeness. 
Hence this foil involves an Invalid Assumption (lA). 


contempt for stupidity and the confining of economic power to a minority 


(lA) except for the arbitrary rule used for classifying foils which 
allows only one foil in a category per item. On the other hand, the 
concentration of economic power for the purpose of maintaining economic 
institutions is a practical necessity independent of how the society 
treats the individual or ho w the decision makers are chosen so that this 
statement is true but irrelevant to the problem. Hence this foil was 
classified as an Irrelevancy (irr). 


Foil 10 (or 1D 9 ) assumes a functional link between co-operative- 



group. For this reason this foil could have been an Invalid A 


t i on 


2. Which of the following is the most important causative 

factor of contempt for stupidity? (Bloom’s Category 4.10) 


b 


In this code used in Chapter V the subscript stands for the 
first distractor (foil) in item 1. The code gives a standard 
procedure for identifying foils without concern for which 
alternative is the right answer. 


*A 0 Compulsory categorising in school, 

B. Compulsory school attendance. (SUB) 

C. Compulsory written examinations. (OS) 

D. Compulsory intelligence testing. (irr) 

Item 2 is asking for the M causative factor” which is not stated 
in the selection. This item, therefore, requires the examinee to 
demonstrate his ’’ability to recognize unstated assumptions” (Bloom: 

1956, p. 205 ) ? hence the classification of this item as analysis (4.10) o 

Compulsory school attendance is an enabling factor in this 
situation, but it is neither necessory nor sufficient. In fact, compul- 
sory school attendance is disjunctively related to the development of 
contempt for stupidity which can develop in universities where attend- 
ance is not compulsory. Also, contempt for stupidity need not develop 
in a compulsory school system. Any attempts at the ins tut i onal i za t i on of 
an individual can lead to the development, on the part of the individual, 
of contempt for the forms of behavior considered "stupid” by that 
institution. (it can also have the opposite effect). In any event, the 
replacement of this term "any” by the term "compulsory” makes this a 
Substitution (Sub) foil. 

With respect to foil 2C (2D^) the cause of contempt for 
stupidity is the "pass-fail syndrome” which compulsorily classifies a 
certain proportion of the population as "stupid.” In this case, it is 
not the written examinations but the use to which they are pat which 
leads to contempt for stupidity. Furthermore, contempt for stupidity 
can (and dees) develop in the classroom context at times other than 
during examination writing by the tacit acceptance by a student's peers 
of the assumption made by the teacher that making a mistake is "sinful.” 


There would be no need for compulsory examinations, if there were no 
attempt to make classifications. However, written examinations by 
themselves do not cause contempt for stupidity, hence this foil is an 
Oversimplification (OS) . 

In the case of compulsory intelligence testing (Foil 2D or 2D^) 
the issue is whether or not the results of these tests are used as part 
of the compulsory classification system rather than whether or not the 
tests are given. Therefore, this foil as stated is an Irrelevancy (lrr) 0 

3. The school acts as an agent for the continuance of contempt 

for stupidity by: (Bloom's Category 2.10) 

Ao Placing too much emphasis on success in extra-curricular 
activitieso (Sub) 

*B. Reflecting the attitude that personal worth is at stake „ 

C. Encouraging competition between students of unequal 
ability. (irr) 

D. Stressing knowledge as the only means to success. (OS) 

Dexter's approach to the schools is essentially upon an 

emotional level. In attributing a person's inferiority complex to the 
school his implication is that the basic strength of contempt for 
stupidity is in its reflection upon self-esteem. This item tests the 
examinee's "ability to understand nonliteral statements (exaggeration) " 
(Bloom: p Q 204). The classification is 2.10. 

In foil 3A (3D^) the phrase "in extracurricular activities" 
would have to be replaced by the phrase "in academic pursuits to the 
exclusion of success in self-corrective activities," to be correct. 

This foil contains a Substitution (Sub). 

Competition between students of unequal ability can lead to the 
continuance of contempt for stupidity provided that the purpose of the 
competition is to make the less able appear stupid. It is the 
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objective and not the fact that is critical. The fact itself is 
irrelevant; therefore, this foil is an Irrelevancy (irr). 

In foil 3D (3D^) the critical aspect of this statement is "to 
the exclusion of success in self-corrective activities." Hence this 
foil is an Oversimplification (OS). Notice that foils 3A (3D^) and 
3D (3D^) are both related to a "correct" answer which is not given in 
this iteiru It would seem perfectly legitimate when more than one right 
answer is possible for a particular item to use alternative right 
answers for the generation of foils. 

4. The author, in charging that ’’society teaches contempt for 
stupidity and a fear of being regarded as stupid” by means 
of the school, is assuming that: (Bloom’s Category 4.20) 

A. The school should not be an enforcing arm for the 
customs of society. (OG) 

*B. The school is a. more powerful socializing force than 
the home. 


C. The home is a more powerful socializing force than the 
school. (inv) 

D. The school is an enforcing arm of the customs of 
society. (irr) 

This item asks the examinees to "recognize a hidden assumption" 
(Bloom: p. 206) which makes this an analysis (4.20) item. 

If the school is at fault it must ha/e more influence on the 
child than the home has on the child. Foil 40 (4D^) must be an 
Inversion (inv) by virtue of being opposite to the correct answer. 

Foils 4A (4D^) and 4D (4D^) are related since both contain the 
same irrelevant premise. However, 4A (4D^) also contains the additional 
unwarranted value judgment "should not be." By virtue of the rule which 
excludes category repetition, it becomes reasonable, at least superfi- 
cially, to classify 4D (4D^) as an Irrelevancy (irr) and 4A (4D^) as 


an Overgeneralization (OG). In the case of 4A however, this 

Overgeneralization is an unreasonable extension of a statement which, in 
itself, is incorrect, suggesting that second thoughts might have led to 
a more reasonable classification of this foil, as the results of 
subsequent analysis showed. 


Second R eading Select i cri 

Source: Marris, Peter; The Experience of Higher Educati on , 

London, Routleage Kegan Paul, 1964, po 175 - 

In this sense, it does not matter what subject a student 
studies, since each is leading towards a generalized intellec- 
tual awareness. But the starting point is still important since 
a student has the greatest incentive to understand whatever 
relates most immediately to his interests. Nor are the concepts 
derived from any one field of study equally relevant to any 
others: the ramification of insights remains biased by its 

roots. The intellectual content has to both guide and be guided 
by the purposes for which a student seeks understanding. 
Otherwise it is meaningless. 

If, then, higher education aims to teach students how to 
abstract, from a particular context, principles by which they 
can organize the perception of their universe of thought, it 
requires that these students have a use for such free-ranging 
understanding. When they enter higher education, their aims 
are confused, and they may not see, or wish to see, the value 
of a generalized intellectual skill. Their approach to learning 
has been conditioned by extraneous motives: they worked to win 

approval or avoid blame, to pass an examination, as much as or 
more than for the sake of understanding. They are not used to 
asking themselves what they want to understand, or why, but 
derive enough interest to master the skills required of them 
from a desire to satisfy the authority who sets the task. So, 

I think, the function of higher education is as much to develop 
the autonomy of their desire to understand, as to satisfy it. 

5. The author suggests that a generalized intellectual 

awareness can be achieved by: (Bloom's Category 3.00) 

A. Focusing on progressively more difficult topics in a 
subject. (IA) 

B. Teaching the students how to generalize from specific 
content. (CM) 
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C. Presenting highly abstract material which is extensive 
in scope. (OS) 

*D. Presenting any subject matter in any predetermined 
sequence . 

This item was treated as an application (3*00) item because it 
requires the examinee to make M use of abstractions in particular and 
concrete situations. n (Bloom: p. 205). For this reason this item was 

classified as application (3.00). The right answer is also, in fact, 
an Oversimplification (OS) because Marris says M the starting point is 
important M (see p. 4?). However, the use of this phrase in 5B would 
have produced a "Clang association 1 ’ which Thorndike and Hagen (l96l) 
point out should not be used 0 (See p. 28) . Alternative 5B? is, 
nonetheless, the most nearly correct of the four alternatives. 

The first foil (5A or 5B^) assumes that l) some specific 
subjects are needed for the development of generalized intellectual 
awareness and that 2) instruction must, of necessity, begin with the 
"easier” topics first. Both of these assumptions are explicitly stated 
as invalid in the selection. This foil was classified as an Invalid 
Assumption (IA). 

The relationship between generalized intellectual awareness and 
inductive reasoning as suggested in 5B (51^) is a vei T common over- 
simplification. With another oversimplification foil in the same item 
the classification of this foil is a Common Misconception (CM) would 
seem to be quite reasonable. 

Similarly, the identification of generalized intellectual 
awareness with the transferability of content is an oversimplification 
of the topic of Marris 1 (1964) discourse. Therefore, foil 5C (51^) is 
classified as an Oversimplification (OS). 


6. The purpose of developing a generalised intellectual 

ava re n ess is to: (Bloom's Ca t e go r y 2*30) 

*A. Promote thinking ability which is not contextually 
bound . 

B. Enable an individual to master any subject area, (OG) 

C. Stimulate thinking ability within the individual's 
chosen field*, (Sub) 

D. Give the individual an ever-widening view of his 
world. (irr) 

in order to answer this question, the examinee is expected to 
make an "extension of trends or tendencies beyond the given data.” 
(Bloom: p. 205)- On this basis this item was classified as 

Comprehension (2.30). 

In foil 6B (oD-j ) the mastery of "any subject area" is far too 
broad a statement for the purpose of Marris' (1964) discussion. Hence 
this foil is an Oergeneraiizaticn. (OG) 

In the case of 6C (61)^) the phrase "within the individual's 
chosen field" is substituted for the phrase in the correct manner 
"which is not contextually bound" making this a Substitution (Sub) 


foil. 


For foil 6 D-( 6 D 2 ) the absence of context in the right answer 
renders the We 1 1 an s hauung (World view) aspect of this foil irrelevant, 
hence 6 D (61)^) was classified as an Irrelevancy (irr). This foil could 
also be a Word-Word Link (WW) because of similarities in phrasiology 
between "ever widening view" and "free ranging understanding" (see p. 


152 ). 


7 o Of the following, the best example of generalized 
intellectual skill is: (Bloom's Category 3*00) 


A. Thinking within the confines of particular subject 
areas. (Sub) 
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B. 

Generalizing from 

the 

con c r e t e 1, 0 t lie ab s t ra c t . 

(ww) 

*C. 

The widely applies 

able 

technique of logic. 


D. 

Applying abstract 

principle to new situations. 

(OS) 


The phrase "best example” in the stem led to the classification 
of this item as an Application (3.00) item. 

There are strong similarities between this item and the previous 
two. For instance, the "particular subject areas” phrase is similar to 
the "individual’s chosen field,” in item 6 so that foil 7 A ( 7 D^) should 
also be classified as a Substitution (Sub). With respect to 7B (7D^) 
the use of induction is similar to pB ( % 5T) ) o In this case, however, the 
Word-Word Link between "generalized” in the stem and "generalizing” in 
the foil is somewhat stronger because of the context than in item 5 * 
Hence 7B (7Dg) was classified as a Word-Word Link.. (WW) 

Also, the confusion in equating transfer of training with 
generalized intellectual awareness found in 5 C ( 51 ^) reappears in 7 D 
(7D ? ) which makes it reasonable to regard this foil as an Over- 
simplification (OS) as well. 


Thir d Reading Selection 

Source: Kagan J. and Moss, Ho A.; Birth to Maturity , 

NoY., Wiley, 1962, p. 85 . 

Aggression is a second behavior system that begins its growth 
during the first five years 0 Traditionally a response was 
labeled aggressive if the goal of the behavior was assumed to 
be psychological or physical injury to a person or person 
surrogate 0 We have adhered to this definition. As with depend- 
ency the display of aggressive acts is a regular concomitant 
of development. The slapping or pushing of an age mate, the 
destruction of a sibling's new fort, and the stinging verbal 
attack are regularly observed in the behavior of many children 0 

Aggression, like dependency, is subject to socialization 
pressures, for the child does not have complete license to 
unleash his anger when he chooses. In addition, as with 


dependency, the occurrence of overt agression is a function of 
both the threshold for motive arousal and the intensity of 
anxiety associated with direct expression of this behavior. 

In contrast to dependency, however, the potential for 
conflict over aggression is greater for females than for males. 
The pattern of social rewards and traditional sex-role standards 
act in concert to discourage the direct expression of aggression 
in girls and women. It might be anticipated, therefore, that 
aspects of aggression would be more stable for males than for 
females. This is precisely what occurred, for overt aggression 
to mother and frequent tantrums during childhood predicted adult 
aggressivity for men but not for women. 

8. If the school were to encourage tolerance for honest 
mistakes, we would expect aggression to: 

(Bloom's Category 4.10) 

A. Diminish somewhato (lA) 

*B. Take different forms. 

C. Disappear completely. (OG) 

D. Remain unchanged. (inv) 

Although this item makes a reference to the first reading 
selection (Stupidity), the question can be answered within its own 
context. For this reason this item was not classified as synthesis but 
as analysis (4.10). The logic of this item revolves around the assump- 
tion of the author that aggression is an innate characteristic of human 
beings which can be modified but not diminished. Thus 8A (8Do ) can only 
be true if this assumption is violated. Foil 8A (8D^) would seem tc be 
an Invalid Assumption (lA) foil in the sense that the examinee must 
make an invalid assumption to select this answer as "correct. M Foil 8C 
(8D,,) strongly overstates the same error as found in 8A (8D-^). For the 
same reason as 4A (4D^) this foil was classified as Overgeneralization 
(OG). 


Changes in the psychological climate w ill lead to changes in the 
modes of expression of aggression which makes foil 8D (8D 0 ) an Inversion 


(Inv) 


I 64 


9o The 'basic position of the author in writing about aggression 

is that it: (Bloom's Category 4.20) 

*A 0 Is inevitable but can be direct through socialization. 

B. Can be eliminated through the process of socialization. 
(Inv) 

C. Will result in internal conflict independent of the 

env ironment 0 ( Sub ) 

Bo Is crippling to the individual by wasting considerable 
energy 0 (CM) 

This item requires the recognition of the assumption of the 
authors which was indicated in relation to item 8 0 For this reason the 
item was classified as analysis (4.20). 

Foil 9B (9B 1 ) is an Inversion (inv) for the same reason as 
8B (8D 3 ). 

Aggression, as an innate behavior system, will produce internal 
conflict only if its modes of expression are frustrated. Hence, 
internal conflict is "dependent upcn environmental conditions" rather 
than being "independent of the environment «, " Since there is already an 
Inversion (inv) foil in this item, another category had to be found for 
this foilo Comparing the two statements "dependent upon.. c " and 
"independent of. 6 ." the latter phrase can be treated as a replacement 
for the former. Therefore, this foil 9C or 9B^) was classified as a 
Substitution (Sub) f 

Aggression can be harmful, but as one basis for intrinsic 
motivation it can be constructive as well 0 Foil 9B (9B^) by treating 
aggression as being exclusively harmful oversimplifies the situation,. 
This oversimplification is so commonly held that this foil was 
classified as a Common Misconception (CM). 


10. With which of the following statements concerning 

aggression would the author be most likely to agree? 

(Bloom's Category 6.10) 

Ao Aggression is like dependency in that it is harmful to 
personality development 0 (CM) 

Be Aggression generally interferes with the attainment of 
educational goals. (inv) 

*Co Aggression is potentially useful for educational 
purpose o 

D. Aggression is considered to be a response to threats 
to a person or person surrogate. (WW) 

This item asks the examinees to evaluate the statements made in 
the alternatives against the information given by the authors about the 
topic, therefore this item was classified as Evaluation (6 o 10). 

Foil 10A (10D ) was classified as a Common Misconception (CM) 
on the same basis as foil 9f (9D.^). 

As pointed out with reference to item 9> aggression can be one 
of the bases for intrinsic motivation. Hence the possibility that 
aggression may be potentially useful for educational purposes may be 
inferred from the selection,. In this case the opposite statement as 
in 10B (lOD^) must be an Inversion (inv). 

Foil 10D (10D^) is best classified as a Word-Word Link (WW) on 
the basis of the phrase "person or person surrogate 0 fl This foil is 
wrong because it contains the phrase "a response to" which is 
extraneous to the definition of aggression 0 (see: p. 51 ) 

11. Overt aggression would likely be decreased by: 

(Bloom's Category 3o00) 

A. Blocking of many modes of aggression. (inv) 

B. Lessening the threat of punishment . (*0S) 

*C. Increasing the threshold of motive arousal. 
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D. Motivating people to rise above their peers. (HT) 

This item was classified as an application (3oOC) item because 
it asks for a practical method of behavior change. 

Blocking of modes of aggression (11A or 11I>^) would be expected 
to intensify responses in the remaining available directions, hence 
this would not necessarily decrease overt aggressiorio This foil was 
classified, therefore, as an Inversion (inv). 

If the threat of punishment is lessened, overt aggression may 
or may not temporarily increas e « depending upon the amount of frustra- 
tion which has previously developed and the v/ay in which the threat is 
lessened. If the release leads to an increased frustration the increas 
in overt aggression would continue. On the other hand, if lessening 
threat also lessened frustration and provided for alternative modes of 
expression, overt aggression could decrease. Foil 11B (llD^) must be 
considered an Oversimplification (OS) in this context. 

Rising above one’s peers can involve the use of aggression as 
intrinsic motivation but it can also be accomplished by the use of 
overt aggression or the threat of its use(i 0 e 0 intimidation). The term 
’’motivating" in this sense refers to "behavior modification" rather 
than motivation in the sense used by psychologists « Foil 11B (llD^) 
was classified as a Redefinition of Terms (RT). 

12o Aggressive behavior in female children is: 

(Bloom’s Category 2.30) 

*A. More likely to produce guilt feelings than in males. 

B. A less likely occurrence than in males of the same 
age. (CM) 

C. More unpredictable and is expressed differently than 
in males of the same age group, (OG) 


D. Less differentiated in expression than in males of the 
same age , ( Inv ) 

Once again the examinee is expected to go beyond the given 
information in order to recognize the role of guilt in child rearing 
practices. For this reason this item was classified as comprehension 

(2o30)e 

Since aggression in males and females tends to take different 
forms because of sex differences in child rearing practices, guilt is 
more likely w r ith females. Both sexes show aggression. The difference 
in mode of expression leads to the common misconception that girls are 
less aggressive than males, hence the classification of 12B (12D^) as 
a Common Misconception (CM). 

The first two words in 12C (12D n ) make this statement false. 

The lack of predictability is from childhood to adulthood and not 
across peer groups. This foil is therefore classified as an 
Overgeneralization ( OG) . 

Foil 121) (12D^) is exactly opposite to the true state of 
affairs making this foil an Inversion (inv). 

13. If we assume that the school increased its use of contempt 
for stupidity as a motivating device, we would expect that 
(Bloom’s Category 5«30) 

A. Parental pressure would intervene to prevent the 
school from making this change 0 (IA) 

*B. Overt aggressive behavior would increase and 
autonomous thinking would decrease. 

C. Both academic success and generalized intellectual 
awareness would increase. ( CM) 

Do The level of student motivation v/ould decrease rather 
than increase. (RT) 

This item is classified as a Synthesis item (5.30) because it 
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involves mere than one reading selection in order to achieve the answer. 


Foil 13A (13D 1 ) involves the assumption that parents would 


oppose this move. However, contempt for stupidity could not he used at 
present if it did not have at least support by implication from parents 
at the present time. The major supporters of the school system, the 
middle class, want to keep the "riff-raff" out of the professions as 
unwanted competition for the aspirations they have for their own 
children. Contempt for stupidity as Dexter (1964) implies is an 
extremely effective method of destroying the academically unfit. It is 
unlikely that powerful parents would oppose this move 0 This foil was, 
therefore, classified as an Invalid Assumption (IA). 

Contempt for Stupidity has the effect of maintaining the 
dichotomy between the academically successful and the others. Increas- 
ing this pressure would sharpen the dichotomy and would not necessarily 
increase the academic success of the survivors and would most certainly 
not increase the academic success of those who did not survive. On the 
other hand, the overt effect of this increase would be to produce an 
apparent increase in academic standards which would be expected to lead 
to the common misconception that making school achievement more diffi- 
cult improves the quality of schooling. For these reasons foil 13C 
(13D 2 ) was classified as a Common Misconception (CM). 


positive intrinsic motivation (i.e. interest). The use of contempt for 
stupidity is, in fact, increasing the level of extrinsic aversive 
motivation. This foil was classif led , .therefore , as a Redefinition of 
Terms (RT) foil. 



redefines motivation in the narrow 7 sense of 
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l4o Which of the following best describes the probable 

relationship between contempt for stupidity and generalized 
intellectual awareness? (Bloom's Category 5° 30) 

Ao Changes in either will have no effect on the other. 
(Irr) 

*Bo As one increases the other will decrease. 

Co Either will increase with an increase in the other. 

( Sub) 

D. Contempt for stupidity should be reduced and awareness 
should be increased. (OG) 

This item, once again, involves two selections (Stupidity and 
Awareness) o For this reason it was classified as a synthesis (5®30) 
item. A person who is motivated by contempt for stupidity (his own and 
others 1 ) would be expected to be constantly en guard e against making 
mistakes o Such an orientation toward his own behavior would tend to 
make him intellectually cautious and hence less inclined to the 
expansive thinking needed to develop a generalized intellectual 
awareness o These two variables would be most likely to be inversely 
related thus explaining the correct answer. 

Treating these two variables as unrelated as in Foil 14A (l4D^ ) 
is contradicted by the information in these two passages. This 
statement could be true in another (independent) context, hence the 
Irrelevancy (irr) classification of this foil:, 

One part of the statement in foil 14C is correct, the 

other incorrect, hence this foil was classified as a Substitution (Sub) 
foil . 


the 

This 


Foil 14D (l4D q ) contains an unwarranted value judgment when only 
relationship and not its psychological importance is asked for G 
foil was classified as an Overgeneralization (OG). 
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The classification of the foils in this item is difficult 
because three of the foils are based to a large degree upon logical 
relations rather than errors in logic. The three possible relationships 
direct (l4D) inverse (l4B) and unrelated ].4A) form the basis for three 
of the four foils. It might have been more reasonable to have 
classified 14A (l4D^) and 14C (14 D^) as Other (0) than to attempt to 
establish classifications on the basis of the rather tenuous arguments 
given here. 


Fourth Reading Selection 

Source: Dinkmeyer, D. Co ; Child Development , Englewood Cliffs, 

N. Jo, Prentice-Hall, 1965, p. 59. 

The social studies committees were working on their reports. 
Doris was chairman cf the southern states committee which 
included Jack, Susan, and Bill. There seemed to be confusion in 
this group so I decided to investigate. "Jack won't co- 
operate," complained Susan. "What do you want him to do?" I 
asked. Jack was frowning. "They say I have to study economic 
conditions in the states, and I am interested in state 
capitals," said Jack. "Did you volunteer to take economic 
conditions?" I asked 0 "There wasn't a chance to volunteer 0 
We were just told her plans," answered Jack. "Is anyone 
investigating the state capitals?" I asked 0 The children 
indicated this job had not been assignedo "In that case, does 
the group mind having Jack study the capitals?" No one seemed 
to care. "What about the rest of you- -are you all satisfied 
with your jobs?" They were. Jack went to the reference shelf 
and started to read. tl lt 

-D . il. 

15* From this report we may infer that the: 

(Bloom's Category 2.20) 

Ac Classroom is very well equipped with instructional 
materials. (OG) 

*Bo Classroom probably has moveable seats. 

C. Class is studying the Southern United States. (WW) 

D. Teacher favours voluntary participation*, (RT) 

This item involves a "reordering, rearrangement , or new view of 
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the material (Bloom: p. 205) which explains its classification (2.20) 

level . 

Foil 15 A (15D ) overstates the situation in the phrase "very 
well equipped" proposing an inference which goes far beyond the 
information in the passage than is reasonable. This foil was 
classified, therefore, as an Overgeneralizati on (OG). 

In foil 15C (15D ) only one committee and not the whole class 
seems to have been studying the southern United States. Since this 
item already has an OG foil, the next most reasonable is a Word-Word 
Link (WW) on the grounds that careless reading might lead to this word 
association 

In foil 15D (15D yj the term voluntary is used in more than one 
sense. Actually, the teacher has substituted her own arbitrary 
decision for Loris' decision This foil could also have been a WW foil 
except that IfC (15D ) fits this category better. For these reasons 
foil 15 D (l^D^) was classified as Redefinition of Terms (RT) 0 

1 6. If the teacher had written "Doesn't work well with others," 
as an anecdotal record for the above incident, this would 
have been: (Bloom's Category 6.10) 

A. Better; it says the same thing with less words, (irr) 

*B. Worse; it fails to indicate the circumstances of the 
incident o 

Co Better; the details of the event are unnecessary when 
judging Jack's behavior. (OG) 

L. Worse; teachers are failing in their obligations in 
not supplying complete information. (CM) 

This item clearly asks for a value judgment based upon the 
evidence in this reading selection which makes it an Evaluation (6.10) 


item. 


Since it should be evident by comparison 


the two alternative 


descriptions given for this same incident that the function of an 
anecdotal record is to give a clear picture of an event for future 
reference, a goal of ’’saying the same things in less words” is 
irrelevant to the task at hand. For this reason foil l 6 A (1611,) was 
classified as an Irrelevancy (lrr ) 0 

Adding unwarranted statements concerning* judgment of behavior 
makes this foil, 16c (^D^), an Overgeneralization (OG). 

Once again, inappropriate value judgments are involved in IoD 
( 161^)5 this time directed at the teacher rather than the student. 

This attitude is so common that this foil was treated as a Common 
Misconception ( CM ) 0 

17. From the above passage we can infer that Doris' leadership 
of the group was: (Bloom's Category 4.20) 

*Ao Coercive o 
Bo Autocratic o (OG) 

C. Destructive. (inv) 

D. Laissez-faire. (Sub) 

In item 17 the examiner is expecting the examinee to ’’comprehend 
the interrelationships among ideas in a passage (Bloom: p. 206)." 

Hence the analysis (4.20) classification. 

On the basis of the argument that the successful autocrat would 
not tolerate contradiction and therefore have no overt objection to his 
or her decisions, Doris' leadership was considered as "coercive" rather 
than "autocratic" for the best answer 0 Notice, by the way, that the 
teacher is a successful autocrat in this passage. For these reasons 
foil 17B (17D ) wa.s regarded as an Overgeneralization (OG). 
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It is not evident from the passage that Doris’ leadership was 
destructive. In fact, she apparently had the support of the two members 
of the committee one of whom reported the problem to the teacher. Being 
opposite to a possible best answer, foil 1?C (l?D^) was classified as an 
Inversion (inv) . 

Since Doris* attempts to coerce Jack were ineffective, she 
permitted the teacher to intervene. As a result, her later leadership 
was laissez-faire, but only under the arbitrary intervention of the 
teacher. The replacement of her later performance for her former 
performance led to the classification of this foil (17D or 17D^) as a 
Substitution (Sub) . 

Once again, the classification of these foils is tenuous and 
open to disagreement . The format of these foils also deviates from the 
usual format of foils in this test, as in the case of item 13 and 
items 19 to 24 inclusive. It is possible that an ’’Other” (0) classifi- 
cation of these foils would have been more reasonable. 

18 0 From the description of the incident, we can conclude that 
the teacher's handling of the incident was: 

(Bloom's Category 6.10) 

A. Good; she intervened to preveiit a serious conflict from 
continuing. (Sub) 

*B Poor; she allowed Jack to use her authority as a lever 
to get his own way. 

Co Good; she resolved the problem to the mutual 
satisfaction of the group. (irr) 

D. Poor; she failed to collect sufficient information 
before enforcing a decision. (OG) 

This item created a good deal of consternation upon its first 
administration At issue seemed to be philosophical differences between 


the examiner and the examinees. 


It is probable that this problem may be 


a complication which is possibly inherent in any multiple choice 
Evaluation (6 C 10 ) item where the evaluative criteria is not supplied. 

The problem arose essentially because many examinees insisted that the 
function of the teacher was to prevent or to eliminate conflict 0 In 
this case either 18A or 18C would be correct depending upon the 
interpretation given to the phrase in the reading selection "no one 
seemed to care 0 ” 

The examiner, on the other hand, took the stand that the 
function of the teacher is to educate. If conflict arises the conflict 
should be used in an educational manner. In this case, the teacher 
should have found cut why the topic of economics was important enough 
to the group to have engendered the conflict. Once Jack understands its 
importance he may agree to do it. That is, the teacher helps to improve 
communication. If, on the other hand, Jack remains adamant, forcing him 
to do something disagreeable to him may not help. In this case, the 
reorganization of committees mere nearly upon sociometric lines might 
improve the situation. There may be some personality reasons for Jack’s 
behavior. In this case, the teacher’s long-term role is to help Jack 
cope with his own and others’ personalities. Letting Jack have his own 
way does not help meet this latter goalo Hence the keyed answer. 

Some of the examinees argued that they had insufficient informa- 
tion to answer this question. This argument was discounted because the 
differences still seemed to be philosophi cal . In any case, the few 
people who chose the keyed answer had the highest average total-correct 
score which meant the retention of this item. 

The prevention of conflict rather than the educational use of 
conflict led foil 18A (18D.^) to be classified as a Substitution (Sub). 


With education as a goal, the mutual satisfaction of the group (as in 


18C or ISDg) is irrelevant, hence the foil was considered an Irrelevance 
(Irr)c 


The classification of 18D (].8P^) as an Overgeneralization (OG) 
is somewhat arbitrary. The teacher could have sought more information, 
but the essential problem is her use of the information which she 
obtained. Since 18C (18I» 9 ) was already classified as an Irrelevancy 
(irr) some other classification is needed. 


Fifth Reading Selection 

Source: Prescott, D. A. ; The Child in the Educative Proces s , 

N. Y. , McGraw-Hill, 1957 ? PP* 125-126. 

Progress Repor t 


X Attendance Area Y County Schools 

Name: Chester M Tea.oher: Miss 0. Grade: 6 

Pays Absent: 0 Pays Tardy: 0 

Reading: Is reading independently on the third-grade level and 

instructionally on the fourth-grade level 0 Poes not enjoy 
reading. Finds many excuses to leave reading to do something 
else. Has trouble understanding what he reads. Is better able 
to find facts than to interpret facts. Has trouble finding 
words in context when meaning is given. 


English Language : Has a wide speaking vocabulary 0 Uses correct 

English. Poes not enjoy story writing. Understands sentence 
construction. 


Spelling: Learns words in spelling lessons and uses them in 

written work. Enjoys spelling. 

Writing: Spaces words well. Is practicing again on the 

formation of letters. Is not neat in written work. Erases 
often. 


Arithmetic: Has worked again this year on addition, subtraction 

and multiplication<> Had some trouble with subtraction,. Is not 
ready for division. Has had experience with problem solving. 
Enjoys arithmetic . 


Social Studies: 


(History, geography, and civics). Has worked 
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with maps • Takes part in discussion. Showed interest in a 
study cf his community. Shared materials. Is trying for a 
better relationship with classmates. 

Science: Experimented with the force of air. Has become 
interested in cloud formation 0 Likes dogs. 


Music and Art: Listens to music. Takes part in singing and 

rhythms. Enjoys all phases cf music. Works with clay, wood, 
paints, and f ingerpaints. Enjoys all media of art expression. 

Instruction for Questions 19 to 24 

Eased on the above progress report answer the next six questions 
by marking: 

A. If the hypothesis is supported by the facts. 

B. If the hypothesis is implied by the facts. 

C. If the hypothesis is refuted by the facts. 

D. If the hypothesis cannot be tested by the facts. 

(Bloom's Category 4.20) 


Hypotheses to test : 


19 


o 


Chester is not liked by the other children; he 
trying to read because he doesn't want them to 
fail . 


avoids 
see him 


20. Chester lacks character. He does all sorts of bad things 
and will not discipline himself to learn to read because he 
has not been punished enough. 


21. Chester is growing very slowly and really is quite immature 
for his grade. Everyone ’ expects too much of him. 


22. Chester has no real reason to want to read, since no one 
ever reads at home. 


23. Chester's reading deficiency has not yet begun to affect 
seriously his performance in other areas « 

24. Chester's mother has kept after him about reading 'until he 
hates it. 


All six of these items were classified as Analysis items (4.20) 
because of the hypothesis testing characteristics of their formate 


This format was used as a marker for analysis subtests. However, 


because of the format, the classification of the foils became 
problematic. The simple expedient was used of classifying all the foils 
for these items as Other (0). 

25« The most useful suggestion to help Chester is: 

(Bloom’s Category 4,20) 

A. To give Chester personal warmth, acceptance and support 
wherever it is appropriate. (Sub) 

*B. To give Chester concrete help in getting started on 
specific tasks, especially in reading. 

C. To give Chester responsibilities and roles of 
acknowledged importance in the daily life cf the 
classroom. (irr) 

D. To try to get Chester’s mother to take the pressure 
off him and offer him more opportunities for self- 
direction. (IA) 

The examinee is expected to comprehend interrelationships in the 
answering of this item, hence its analysis (4.20) classification. Foil 
25 A (2533-^) substitutes emotional support for corrective instruction, 
hence the Substitution (Sub) classification of this foil. 

The treatment suggested in 25C (25D^) has no bearing on his 
academic needs; it was therefore classified as an Irrelevancy (irr). 

There is no evidence in this selection that there is an 
unreasonable pressure on Chester by his mother; hence this foil 25D 
(25D^) involves an Invalid Assumption (IA). 

26. If additional information on Chester is desired, and none 
of the following had been attempted, which one would 
provide the greatest amount of immediately useful 
information? (Bloom's Category 5*20) 

Ao An interview with Chester's previous teacher. (0) 

*B. An interview with the parents. 

C. A diagnostic test in reading skillso (OS) 
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Do A request for the assistance of a guidance counselor. 

(Irr) 

This item involves the examinee generating a structure to 
represent Chester’s entire situation by inductive reasoning prior to 
answering the question 0 For this reason, this item was regarded as a 
synthetic (5 .20) item. 

The best first hand source of information about Chester is his 
parents. The next best is his previous teacher. Since the Guidelines 
do not make any provision for this kind of relationship, foil 26A (26 jD^) 
is best classified as ’’Other” (0). 

A diagnostic test in reading skills is only useful to a teacher 
who know enough about these tests and the reading problems they 
diagnose to be able to use them effectively « Also, administering and 
interpreting such tests can be time consuming,, It is an Oversimplifica- 
tion (OS) for foil 26c (26D^) to suggest this course of action to be 
superior to any other. 

Most of the examinees were experienced teachers, hence it was 
reasonable to assume that they would know from experience that guidance 
personnel rarely can give a teacher information they do not already 
know,, This effect occurs because test batteries are rarely more 
reliable than a month or two of sensitive observation by a teacher. 
Therefore, the course of action suggested in foil 26D (26lh) is 
classified as Irrelevancy (irr). 

27. A reasonable conclusion which can be drawn from this 
report is that; (Bloom's Category 4.10) 

Ao Chester's problem stems essentially from his poor 
relationship with his mother. (IA) 

B. Chester's problem sterns essentially from his poor 
relationships with his peers. (irr) 


*C. Chester* s problem has no single cause and no simple 
solution. 

Dw Chester’s problem stems from such a wide range of 

sources that a classroom solution is impossible 0 (CM) 

This item is somewhat difficult to classify because the ’’drawing 
conclusions” is not part of Bloom’s Taxonomy . However, this item also 
involves the examinee’s ’’skill in distinguishing facts from hypotheses” 
(Bloom; p. 205), hence the analysis (4.10) classification of this item. 

Once again, the mythical poor relationship with his mother is 
introduced in foil 2? A (27D^). In this context the most reasonable 
classification of this foil would be an Invalid Assumption. (IA). 

Chester’s problem seems to be centered upon his reading 
difficulty,. His relationship with his peers may influence his motiva- 
tion to attempt improvement, but is irrelevant to his problem. There- 
fore, foil 12B (12D 0 ) was classified as an Irrelevance (lrr) c 

Foil 2?D (27I) 0 ) overgeneralizes to an extreme level which made 
the most reasonable classification for this fo:il to be Common 
Misconception (CM) . 

28. In Chester's progress report, which one of the following is 
the most important factor contributing to his difficulty 
with school achievement? (Bloom’s Category 5*30) 

*A 0 Aggression which is building up due to frustration 
ever his reading development. 

B. His inability to develop a generalized intellectual 
awareness. (irr) 

C. His i^eakness in reading which is affecting all areas 
of learning. (OG) 

D. The teacher has been using ’’contempt for stupidity” as 
a motivating device. (IA) 

This item, once again, involves more than one reading selection 
and was therefore treated as a synthesis (5*30) item. 
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Foil 28B turned cut to be somewhat unreasonable because the 
author assumed that the examinee would be able to identify the fact that 
generalized intellectual awareness is an adult phenomena 0 The evidence 
is tenuous since the Awareness selection from its title is related to 
university education, and Chester in this (Progress) selection is in 
Grade six. This foil would be an Irrelevancy (irr) but it may have been 
unreasonable to expect so tenuous a connection to be made if it could 
not be assumed that the examinees would know this fact. 

The progress report indicates that there are some areas of 
Chester's development which are not out of step which makes foil 28C 
(28D 0 ) an Overgeneralization (OG). 

Foil 28D (28D^) suggests that this teacher is using contempt for 
stupidity for motivation, which may be an Invalid Assumption (lA) since 
the tone of the rerort is sunnortlve rather than condemnatory. 


29* On the basis of the foregoing which of the following seems 
.to be the most important consideration when preparing 

anecdotal records or progress reports? 

(Bloom's Category 5.30) 

A, Make no attempts at interpretation since your judgments 
are probably biased. (CM) 

*B. Present as much information as possible about all the 
salient aspects of the situation. 

C. Be as brief as possible, giving no information which 
may cloud the central problem. (OS) 

D. Do not put anything into these reports which might 
antagonize the child's parents 0 (RT) 

This item also involves more than one reading selection and 
therefore it was classified as a synthesis (5*30) item. 

In the case of foil 29A (29D^), it is impossible to avoid 
judgments in any reporting, hence this advice is a Common Misconception 
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(CM). Because of the problem of observer bias, as much pertinent 
information as is possible should be supplied so that alternative 
interpretations can be considered by others 0 This latter statement 
makes foil 29C (29D^) an Oversimplification (OS). 

In foil 291) (291)^) the term report referring to "progress 
report" is redefined by the suggestion that such a report may become 
public property, i.e, it will be part of the "report card" to the 
parents. This approach involves a Redefinition of Terms (RT) 0 

30 • The most important principle illustrated in this set of 
questions is that the teacher should: 

(Bloom 1 s Category 5,30) 

A. Promote generalized intellectual awareness in 

aggressive children by using contempt for stupidity 
as a motivating force, (WW) 

*B. Recognize that developmental deficiencies arise from 
complex circumstances, requiring multiple-strategy 
o olnt i ons * 

C. Seek professional assistance from the school counselor 
in the identification of developmental problems. (Tr) 

Bo Recognize that "contempt for stupidity" is not 

necessarily an effective way of generating motivation 
in pupils. (RT) 

This item synthesizes the previous twenty-nine which led to its 
synthesis (5*30 ) classif icati on. 

Foil 30A ( 30D-j ) is a good example of a contrary-to-fact 
statement developed by the glib use of the repetion of phrases from the 
reading selections. It illustrates very well the way in which Word Word 
Link (WW) foils might be generated. 

The discussion concerning the role of the counselor which 
occurred on page 66 suggests that it would be more reasonable to get the 
teacher to identify the problem and then get the professional 1 s help in 
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its treatment. However, this approach means changing the role of the 
counselor from diagnostician to prognostician. The relationship 
recommended by foil 30C (30^) may, therefore, be reversed to the ideal, 
and this foil was classified as a Transposition (Tr) for these reasons. 

One of the important characteristics of the so-called "Puritan 
Ethic" is its heavy reliance on adversive extrinsic motivation (i.e. 
punishment) as a means of regulating behavior. Contempt for stupidity 
is an extremely effective method of motivation in this context if the 
ill effects of aversive motivation are ignored. Furthermore, there is 
a tendency, for political reasons, as already suggested that the 
strongest supporters of the school also support this method of motiva- 
tion. On this basis, foil 30D (30D^) is clearly incorrect. For it to 
be considered correct, the term "motivation" must be confined to 
intrinsic positive motivation (or interest) making this foil a 
Redefinition of Terms (RT). 

In general terms, the overall classification of the foils as 
reported here is probably open to considerable disagreement as suggested 
by the low interrater reliability. Would other researchers have 
produced superior results? One of the raters of the items made the 
following alternative classifications: 

1. 6D 0 was classified as a Word Word Link (WW). 

2. lOL^ was classified as the right answer. 


3. 15D 1 was classified as an Invlaid Assumption (IA). 

In this case his classification of 6D^, turned out to be 
supported by the cluster analysis (see: p« 77); his classification of 


10D^ was incorrect for the reason given (see: p. I 65 ) ; and his classifi- 
cation of 15 D ^ was not supported by the cluster analysis. (see: p. 77)* 
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This success ratio of one out of three is equivalent to that of the 
experimenter (see: p c 75) « 

It should also he noted that the nature of the examination was, 
at least in part, predetermined by the examiner’s philosophy of 
education. This characteristic of the examination is evident in the 
first place in the nature of the reading selections upon which the 
examination is based. Second, it is evident in the nature of the 
questions asked concerning these reading selections. Third, it is 
evident in the reasons given for the classification of items and foils. 
In general, it is hoped that the major portion of this bias is confined 
to the nature of the reading selections used and that once these are 
given the astute reader should be able to infer the bias, and answer 
accordingly. The possible exceptions which are clearly evident are 
item 18 and item 28, particularly with respect to foil 28b (28D^). 

It is being argued that bias is unavoidable and, hopefully, can 
only be minimized in its adverse effects upon student performance. The 
clear thinking student should be able to recognize and adopt a number 
of points of view concerning any particular subject matter and apply 
logic, once the point of view is assumed, in order to arrive at 
reasonable conclusions. So long as the logic which follows from the 
assumptions cannot be faulted the system itself can remain intact. 

The purpose of presenting the development of the experimental test in 
such detail was to expose the logic of the test including the reasons 
why the foils are considered to be wrong ( i . e c its construct validity) 
to such reasonable attacks as may be made. If the construct validity 
of the test is supported, on both logical and evidential grounds, then 
the experimental test may be regarded as an effective measuring 
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instrument of higher mental processes independent of the assumptions 
upon which the particular items are basedo In this case the refutation 
of these assumptions by subsequent evidence will not diminish the value 
of this procedure as a method for the construction of measuring 
instruments for the evaluation of student performance in the cognitive 
domain (i«e 0 in the use of higher mental processes) « 
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LOGICO- SEMANTIC ANALYSIS OF RIGHT 
AND WRONG ANSWER CLUSTERS 

This appendix presents a detailed discussion of the items and 
foils which differed in their advance classification from the classifica- 
tion of the clusters in which they occurred,, A cluster was classified by 
the most frequently recurring advance classification in the cluster,, 

The findings for this part of the study were summarized on pages 
70 to 71 for the right answer clusters and on pages 71 to 79 for the 
wrong answer clusters. This detailed analysis is given here for two 
reasons. First, it was felt that the effective reclassification of 
alternatives represented evidence in support for the multiple interpreta- 
tion hypothesis. Second, it was felt that subsequent researchers might 
find value in an independent evaluation of the logic which led to the 
conclusions this study has presented. 

The Meaningful Interpretation of Item Clusters 

In an exploratory study into a new area of research, the 
relative relevance of characteristics can be expected, in general, to be 
unknown. For this reason, the failure of the advance classification of 
items to provide much assistance towards a meaningful interpretation of 
the data was disappointing, but not surprising. 

On the other hand, the cluster solution used, replicated the 
advance classification by Bloom's Taxonomy to the extent that 40 per 
cent of the items which appeared in a single cluster also held a common 
advance classification. Table II (see: p. 73) gives the clusters from 

Group A and indicates which items were in a common classification, the 
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classification of these items, and the final interpretation given to 
each clusterc 

Where the interpretation remained ambiguous in Table 11 

(see: p. 73) the uncertainty is indicated. It would be a fairly simple 

matter with clusters like CL and C 10 , for instance, to assume that the 

o xu 

advance classification of these items adequately interprets the cluster. 

In other cases such as CL, CL or CL, the majority of the items 

1 5 ( 

were in a common class. If the class of the majority is used as an 

interpretation, the members which did not share this common 

classification must be explained 0 

Finally, there were several clusters, CL, CL, C, , CL, and CL 

£ j 4 o y 

which did not contain even two members which shared a common advance 
classification® The interpretation of these clusters would seem most 
problematic, but must be attempted. 

Several considerations were used in an attempt to arrive at an 
unambiguous meaningful interpretation of each cluster. The first and 
most obvious one was the advance classification of the items by Bloom's 
Taxonomy . 

Second, the possibility that some clusters ( CL for instance) 

7 

might be content clusters could not be entirely discounted. 

Third, the possibility that items might have been misclassif led 
in either of two possible wayso The aspect of the item which leads to 
the misclassif ication might be related to some obvious but irrelevant 
format characteristic. This problem has already been illustrated by 
the case of items 19 to 24 inclusive, -(see: Appendix B pp. 175-177)* 

Alternatively, there may be a discrepancy between the way in which the 
examiner intended that the item be interpreted and the way it was, in 
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fact, interpreted by the examinees a For instance, an item which is a 
comprehension item for some students may well be an analysis item for 
others. This latter possibility would suggest that the classification 
of items might be better after their characteristic clustering has been 
determined than before the test is given. 

Fourth, since this study is postulating that the foils may have 
some effect upon the 'interpretation of the item and, therefore, the way 
in which it is answered, the nature and selection ratio of the foils 
should also be taken into account when an attempt is being made to 
interpret the clusters 0 In this respect, foils with a selection ratio 
of o 05 or less were dropped from this and subsequent analyses, since 
these foils were selected by too few people for the statistics pertinent 
to these foils to be stable. 

Finally, there were other sources of information about these 
items, such as the interitem phi correlation coefficient matrix and the 
item consistency which might have proved useful in the attempt to make 
an unambiguous interpretation of each of the item clusters. 

In the discussion which follows each of the ten item clusters 
are dealt with in turn in an attempt to establish an unambiguous 
interpretation for each cluster. In advance of each discussion a table 
appears which supplies the following information: 

1. The numbers of items in the cluster, 

2. The subject matter content from which each item is drawn . 

3o The advance classification of each item. 

4 . The biserial correlation (r_ ) of each item with the total 

v b 7 

test score. 

The difficulty of the items 0 


5 . 
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6. The selection ratio for each foil ana the advance 

classification of each foil for the items in the cluster. 

The foils which are dropped are also indicated. 

Any other information needed for the discussion is supplied in 
the context. Table 42 follows on page 190 • 

Three of the four items in Table 12 have their content clearly 
drawn from the Stupidity reading selection on pages 4l and 42, and the 
fourth one, item 8, has a reference to this selection in its stem. 
However, item 8 can be answered without having read this selection since 
a good student should be able to infer v/hat is meant by the phrase 
"contempt for stupidity” from the context. Also, foil 28 ]j q was the only 
part of item 28 v/hich contained a reference to this selection but, once 
again, it should be possible to infer the meaning from the context. 
Furthermore, foil 28D^ is classified as an IA (invalid Assumption) foil, 
the invalid assumption for v/hich can be arrived at without reference to 
the "Stupidity” selection. In addition, this cluster does not exhaust 
the items in which a reference to this selection is made. There are 
five other such items. For these reasons, it is not possible to 
interpret this cluster unambiguously on the basis of content. 

The relative magnitude (.378 to .456) of the r^ (biserial 
correlations with total scores) varies sufficiently to suggest that they 
were probably not related to the statistical artifacts which caused 
these items to form a cluster. Also, since none of the difficulties or 
selection ratios (D X J are large enough that these items must (of 
necessity) overlap, the ratios cannot be considered to be relevant in 
this event either. 

Since three of the items were classified in advance as analysis 
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items, whereas the fourth one (item 28) is Synthesis as this class was 
defined, the reasonableness cf retaining the Analysis classification for 
this cluster is greater than for changing it to a content-oriented 
classification. Also, as has already been noted, item 28 has identical 
foils by class with item one. Furthermore, in three of these items 
(l, 8, 28) the most commonly selected foil has an advance foil classifi- 
cation of OG (Overgeneralization). In this case, if 2IL could be 
reasonable reclassified from Substitution to OG an alternative to the 
reclassification procedure is to suggest that the categories of foil 
given above may not be independent. In this case, the common element to 
these items would be the common classification of the most commonly 
selected foil. This argument would be strongly supported if all of 
these foils fell into a common wrong answer cluster, which they did not 
do (28D 0 is the exception). It is reasonable to be reluctant to 
classify a cluster derived from the right answer correlation matrix of 
Group A on the basis of the performance of foils to the items when 
performance on specific foils is not part of the statistical basis from 
which this cluster is derived. In this cluster, the classification was 
(reluctantly) retained as Analysis, even though this meant reclassifying 
item 28. Table 43 follows on page 192. 

To begin with, in Table 43 there was no consistency between the 
items in Cluster 0 o with respect to their content (information back- 
ground) which might have accounted for the formation of this cluster. 

A similar statement can be made for the advance item classification, for 
the relative ma'gnitude of the r^ and the coefficients, and for the 
advance classification of the foils. 

Superficially, then, there would seem to be no basis for the 
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interpretation of this cluster. However, an examination of the foil 
classification for item 30 is revealing (see: pp. 181,182); and 301^ was 
an RT (Redefinition of Terms) foil. Two and possibly all- three of these 
foils are related to Comprehension- type operations (i.e. they are 
Mi sread i ng type foils). If the foils of an item could be eliminated by 
comprehension- type strategies, the fact that the stem-right-answer 
relationship involved a synthesis- type relationship may have been 
irrelevant. Similarly, if the stem-right answer relationship can be 
recognized by comprehension-type strategies without having to eliminate 
foils, an item may be a comprehension item with high level foils . Such 
a combination of arguments could account for item 3 and item 30 occurring 
in this group. In order to account for the presence of item 17 in this 
cluster, it is necessary to suggest that the GG, 03, etc., type foils 
are related to analysis-type strategies. For some individuals, then, 
item 3 could have been treated as an analysis item because of the high 
level of the foils. In this case, item 17 would have to have most of 
the examinees who selected the correct answer in the top one- third of 
the group as defined by total score correct, and there would have to be 
a high phi coefficient between items 3 and 17- The results support this 
contention. In the first place, about 80 per cent of the individuals 
who answered the item correctly were in the top 40 per cent of the 
group. In addition, the phi correlation coefficient between item 3 and 
item 17 is .281 which is significant at a probability level of 

.02 > p >.01 o 

These results did not make possible the unambiguous 
interpretation of Cluster C ?0 On the contrary, the interpretation would 
seem to be that this cluster involves multiple strategies, some at the 
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comprehension level, and some at the analysis level. Hence cannot 
be defined in terms of a unitary category from Bloom 1 s T axonomy 0 

Once again, the data suggests right- wrong answer interactions. 
In addition, there appeared to be a multiple-strategy level involvement 
in the cluster 0 

Cluster C4, Table 44 (see: p. 195) proved to be somewhat 

similar to in that the same phenomena occurred once again e All the 
initial bases for interpreting this cluster failed to provide for an 
unambiguous decision. In addition, the high level (Synthesis) item 
seemed to be lowered by virtue of the low level foils. 

It may be reasonable to relate items 4 and 6 because of the fact 
that 4D^ was reclassified a CM (Common Misconception) in the interpreta- 
tion of the wrong answer clusters and 4L^ became unclassif iable . It is 
possible that these are both low level foils which might lower the 
analysis classification of item 4 to comprehension. This argument 
concerning the reclassification of item 4 must remain inconclusive since 
a classification of 4D^ was not established. 

The fact that these two clusters (C and CL) proved to be 
similar raises the question as to why they did net form a single 
cluster. Obviously, the items from one cluster did not correlate 
highly with the items from the other, but this fact does not add any 
information since it is this fact which is the statistical basis for 
the information of clusters. A look at the wrong answer clusters 
(indicated by W ) into which the foils fell proved illuminating. 

Table 45 (see: p. 196) shows that there is a degree of 

similarity within of the wrong answer clusters (W^ occurs in two 
items). Cj has a high level of similarity among the foils (W^ occurs in 
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two items). C 0 has a high level of similarity among the foils 
(W 0 occurs in all three items and VL in two ) « There are no common 

3 ( J 

wrong answer clusters between the two groups * This evidence would have 

been much more conclusive as to why these clusters were distinct if the 

foil groups in were more strongly similar. Once again, however, 

there is some indication that foils may influence the formation of 

right answer clusters. 

In any event, the fact that there seems, once again, to be 
multiple strategies involved in this cluster means that it cannot be 
unambiguously classified 0 


TABLE 45 

WRONG ANSWER CLUSTER MEMBERSHIP 
WITHIN AND BETWEEN RIGHT ANSWER 
CLUSTERS C Q AND C Q 


Item 

No. 

C 2 

Wrong Answer 
Clusters 

1 f 
i i 
t ! 
i i 
t l 
i i 
i t 
t i 
! 1 
l 1 

Item 

No. 

C 3 

Wrong Answer 
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1 ! 
1 1 
t 1 
1 I 
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W 1 
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9 

1 ! 
1 ! 
! 1 
1 t 
• 1 

13 

• W 3 

W 7 
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W 15 W 4 

1 1 
1 1 
1 ! 

1 ! 

! 1 

LJ 
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W 3 

W 11 
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As Table 46 shows (see: p. 198), there is no clear basis for 

the unambiguous classification of Cluster from any of the sources of 
data being usedo This cluster, therefore, remained unclassified* 

On the other hand, item 5? in addition to involving a practical 
application, also involves “going beyond the given data, to determine 
implications 0 o * which are in accordance with the conditions described 
in the 1 original communication 1,1 (Bloom 1 s Taxonomy , 19.56, p c 205). This 
could mean that the best Bloom 1 s Taxonomy classification for this item, 
if the application aspect is ignored, is comprehension (2o30: 
Extrapolation) o This item may, therefore, be capable of a dual 
classification. Item 19 is the only one in this series in which the 
correct answer involves the implication of the statement by the reading 
selection® Item 14 also involves extrapolation except that the extrap- 
olation is from two selections rather than one, making this item 
(arbitrarily) a Synthesis item. It is too early in the development of 
this testing technique to be dogmatic on a post hoc basis about the 
interpolation on a strategy basis , of any cluster 0 This statement is 
particularly reasonable since this clustering does not cross-vaiidate 
(see: p 0 9^) • However, it does suggest that a better definition of 

the multiple strategies which may be involved in the answering of 
multiple choice items for the types represented on this test might 
improve the effectiveness of the advance classification of the item, 
and most certainly would improve the interpretation of the clusters 
which were found to be peculiar to a particular group of examinees. 

In cluster C , Table 47 (see: p Q 199) , the common element, on 
the basis of the decision rule that the cluster be identified by the 
most frequent process category from the advance class, would be 
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is for the correct answer. 
X means foil is dropped. 
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TABLE ^7 

SUMMARY OF DATA FOR CLUSTER C 
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D^ is for the correct answer. 
X means foil is dropped. 
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Analysis. Item ? was originally classified as an Application item 
because the stem asked for "the best example," In other items the 
lowering of the level of performance of that item by low level foils 
was observed. In this case the popularity of the analysis-related OS 
(7D^) foil may have had the opposite effect. Furthermore, foil 7D^ 

(an OS foil) was classified during the interpretation of the wrong 
answer clusters as NS (Non Sequitur) (see: p 0 225 for definition) which 
was one of the three new foil categories which came out of these 
discussions. Since both of these categories (OS and NS) seemed to be 
more related to the logic of the item than to its semantics, these types 
of foil may help to define analysis type items. This possibility was 
strengthened by the fact that about 73 per cent of the examinees who 
chose 7bo (the WW foil) were in the bottom 60 per cent of the group, 
whereas about 65 por cent of the examinees who chose 7b^ (the OS foil) 
were in the top 60 per cent of the group. These figures suggested a 
moderate but definite trend on the part of these foils to move the 
performance of this item upward in level. The same trend is evident in 
the average total-correct values for each of these foils and for the 
right answer. Those who chose 7^^ bad an average total correct score 
of 11.6, while those who chose 7D^ had an average total correct score 
of 12.2, and those who chose the correct answer had an average total 
correct score of 13<>7. The average score on the entire test was 12.2. 

The basis for the upgrading of item 7 to the Analysis level is 
somewhat tenuous. The fact that it would otherwise be the only anomaly 
in this cluster strengthens the use of the decision rule. The use of 
the rule is further strengthened by the fact that none of the possible 
bases being used for interpretation give a more reasonable explanation 
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for this cluster. The Analysis classification of this cluster was 
retained. 


Cluster Cg, as summarized on Table 48 on page 262, was 


classified as Analysis since both its members were so classified in 
advance. However, the procedures being used suggest at least two other 
bases for interpreting this cluster. First, the difficulties (D 
selection ratios) were high. Second, the most commonly selected foils 
fell into the same wrong answer cluster (W^). These two events could 
be related to each other since the advance classification of the foils 
with their respective items are different. 

In any event, was treated as an Analysis cluster in 
subsequent statistical analysis. 

In Table 49 on page 203 none of the bases being used for 
interpretation assist in the explanation of the formation of cluster C 
except the arbitary rule that the majority of the items shared a 
common classif icaticn in advance. In both these items (item 10 and 
Item 16) a value judgment is specifically asked for in the stem and 
explicitly included in each of the alternatives. The other item 
(item 18) which had these same characteristics fell into another 
cluster. Item 29 asks for "the most important consideration” in the 
stem which makes the stem contain an explicit request for a value 
judgment; however, this value judgment is not explicit in the alter- 
natives. The term "likely” occurs in the stem of Item 11 suggesting, 
perhaps, an implicit value judgment may be involved in this stem. 

Should the definition of the Evaluation level of items as used in this 


7 


study be extended to include implicit as well as explicit value 
judgments? The results of the cross-validation part of the study 
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TABLE 49 


SUMMARY OF DATA FOR CLUSTER 
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suggested that Group E responded quite differently to these i terns * 

In Appendix B where Item 26 from Cluster 0 o in Table 50 

o 

(see: p Q 205) was discussed (see: pp 0 .177 >178) it was pointed out that 

an inductive s tincture had to be generated in the mind of the examinee 
in order to answer this question, led to its classification as a 
Synthesis item. All other Synthesis items had the additional 
characteristic of involving more than one reading selection. The use 
of the device of having more than one reading selection as a basis for 
Synthesis items did not generate a unique cluster. On the other hand, 
if the logic of Item 12 and Item 20 are examined, it becomes evident 
that a similar process of reasoning to Item 26 may have been involved 
in these two items as well. 

The classification of Item 20. as Analysis v/as made because it 
is clearly structured so as to involve a hypothesis testing procedure. 

If each of the alternatives in Item 12 and Item 26 were also regarded as 
hy potheses which were to be tested against the inductive structure 
employed by the examinees in Group A, the answering o.f these items may 
have been relatively homogeneous. 

This cluster (Cg) is classified as Synthesis on the grounds used 
for the advance classification of Item 26, or it could be Analysis on 
the grounds used for the classification of Item 20. However, in 
multiple choice format it is impossible to avoid hypothesis testing 
aspects of a Synthesis item when a specific set of alternatives is given 
in the item c The type of item in this cluster came about as close to a 
Synthesis level as it may be possible to come in multiple choice items. 
Bloom (1956) suggests that if ambiguity of classification occurs, it 
should be resolved in favour of classifying to the highest possible 
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D„ is for the correct answer. 

* 

X means foil* * is dropped 0 


level. For this reason, on the oasis of the performance of these 
items (at least within Group A) the items in this cluster have been 
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reclassified as Synthesis items. 

Both 'Cluster C items, as summarized in Table 51 (see: p. 207) , 

are from the same reading selection. Also, the Procrustes rotation to 

content suggested a possible content basis for this cluster. However, 

both of them also proved to be very poor items on the basis of the 

combination of their r and difficulty (D_^ selection ratios) „ In 

addition to this their most common foils occurred in the same wrong 

answer cluster (W 0 ), and the D selection ratios of these two foils were 
v 7 n 

very higho It is probable that these two items form a cluster, at least 

partly, on the basis of the relationship between these two foils. This 

cluster remained unclassified in subsequent statistical analysis. 

There would seem to be two possible bases for the interpretation 

of Cluster C as summarized in Table 52 (see: u. 208) content and 

10 J 

advance classification. It is possible that both of these factors were 

operative to make this cluster distinct from other Analysis clusters. 

Some support for this argument may be found in the fact that was 

positively correlated with all the other classified clusters in the 

Procrustes rotation analysis (see: Table 10, p 0 6 9) except C^, although 

all of these correlations may have been too small to have much meaning. 

In summary, then, this interpretation attempt upheld the 

Analysis level classification of Clusters C n , C., C^ , and C nn as they 

1 5 ° 10 

emerged in the comparison between the results of the minimal interpoint 
distance cluster analysis and the advance classification of the items. 

It similarly upheld the classification of C n as Evaluation The 
procedure led to the reclassification of one cluster (C~) from 
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is for the correct answer. 
X means foil is dropped. 


TABLE 52 


SUMMARY OF DATA FOR CLUSTER C 1Q 






t ! 
! ! 
J ( 


Ra 

tios 


! ! Class! 

I t 

fica 

tion 

Item 

Item 

Advance 

r b 

! ! 
1 1 

D f 1 

R 

D 2 

D 3 

li h 

£ 

T) 

^3 

No. 

Content 

Classification 

! ! 
i : 

! ! 
! 1 
t ! 



I I 
t 1 


21 

Progress 

Analysis 

.404 

t 1 
! ! 
; t 
1 ! 

! f 

,32J I 

.07 

,37 

.23 

1 t 

! ! o 

j i 

0 

0 

25 

Progress 

Analysis 

.329 

1 I 
1 1 
! ! 

! ! 

CLC 1 ! 

0 J J | t 

o 24 

• 19 

.02 

; | sub 

Irr 

X 

24 

Progress 

Analysis 

.317 

! ! 

! f 
! ! 

! 1 

,_1 !_ 

.86 1 j 
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.06 
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! ! 
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LJ 

0 
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Selection 


Advance Foil 


D^ is for the correct answer. 
X means foil is dropped. 
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■undetermined to Synthesis 0 In the remaining: cluster’s (C 9 , 0y 0^ , and 

C^) it was impossible to provide an unambiguous basis for classifying 

these clusters from the available data® Hence these clusters remained 

unclassified© Clusters 0 o and 0 o seemed to involve some form of multiple 

^ j 

strategies and seemed to be a Comprehension level cluster for Group A 
if the superficial characteristics of the items which led to their advance 
classification were ignored© Since this cluster did not reappear in Group 
B, multiple strategies between groups of examinees may be involvedo 
Only in the case of C^ can content be said to be a more 
reasonable interpretation of these clusters than some form of single or 
multiple strategy© Even in this case, the content interpretation is in 
some doubt, suggesting that this test is essentially ’’process oriented,” 
as intended o 


From the original 30 items on the test 12 items (40 per cent) 
formed at least pairs in the clusters which emerged. This ’’interpret- 
ability” figure was improved to 19 items (6 3 per cent) on the basis of 
the interpretation procedure used c 


The Meaningful Interpretation of Wrong Answer Clusters 

A similar interpretative procedure was used for wrong answers 
as for the i terns © The considerations were taken in the following order, 
l) advance classification, 2) information background (content), 3 ) 
statistics of foils, 4) logico-semantic analysis. A cluster was again 
assumed to be identified by the most frequently recurring advance 
classification o Foils of the 0 (Other) category were assumed to be part 
of this common classification. 

Of these four, the only one related to the logico-semantic 
characteristics of the foils was their advance classification. If these 
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clusters could not be accounted for in other ways, and the logico- 
semantic characteristics of the foils can be shown as a possible basis 
of cluster membership, then this latter basis may be the best available 
interpretation. It has already been shown that the advance classifi- 
cation of both right answers and foils did not survive very well in the 
cluster analysis. It has also been shown that this classification can 
be improved by examining the clusters for seme relatively unambiguous 


basis for interpretation. In the case of foils. this involved a re- 
examination of their logic-o- semantic structure. The five bases used in 
an attempt to interpret the wrong answer clusters were, once again: 

1. The advance classification of the foil. 

2. The content of the foil. 

3. The selection ratio of each foil. 

M . The relationship between right answer and wrong answer- 


clusters . 

5» The reconsideration of the logico-semantic structure of the 
foils . 

In some cases, material from other sources such as Powell and 
Isbister ( 1969 ) were used to assist in this interpretative process. 
Finally, one cluster (W^) was completely lost by virtue of the lo w 
selection ratio among all of its members. In addition, two other 
clusters were reduced to single members by this procedure. In the 
discussions which follow some tentative attempts are made to account for 
the "special cases" as well as the "general trends" in each cluster. 

For the convenience of the reader, whenever a particular foil is 
being discussed for the purpose of reclassification, the stem of the item 
and the particular foil are both given ahead of the pertinent discussions. 
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Table 53 on this page gives information for the .interpretation 
of wrong answer cluster W , 


TABLE 53 

INFORMATION FOR THE INTERPRETATION 
OF WRONG ANSWER CLUSTER W 


Foil 

Content 

Advance 
Cla s s if i cation 

D a 
n 

1D 1 

Stupidity 

0G b 

CO 

00 

2D 1 

Stupidity 

Sub 

.61 

22D 3 

Progress 

0 

.84 

8D 1 

Aggression 

OG 

.58 


Li sci pline 

CG 

.80 

X 




a. The symbol 

L refers to the 
n 

Selection ratio. 


b. Foils which 

had appeared in 

a common category in the 

advance foil 


class if icati on. 


The content from which these foils in wrong answer cluster \I 
are drawn comes from a broad spectrum of the test ruling out content as 
a possible basis for the interpretation of this wrong answer cluster. 

The phi coefficients upon which this cluster is based are 
dependent upon the size of the overlap between particular pairs and upon 
the marginal totals 0 When the selection ratios for two alternatives 
both exceed .50 there is tendency for the range of phi to be shifted 
positively o In this case, all of the selection ratios in the cluster 
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of the selection ratios could 


exceed .JO and for this reason the sizes 
be a contributing factor to the statistical formation of this cluster. 

If this event were the only factor, however, it would be reasonable to 
have expected more of the other six foils which have selection ra.tios of 
greater than . 50 in this cluster, or in a limited number of other 
clusters. In fact, they occurred in four of the clusters. 

Of the five foils in this cluster three of them were in items 
which occur in a common right answer cluster (fj^). This finding was 
suggestive, once again, of a right -answer wrong-answer interpretation, 
but insufficient to lead to an unambiguous interpretation of the present 
cluster under discussion 0 

Also, three of the five foils in Table 24 were classified as OG 
(Overgeneralization, i.e. 1D^, 8D^, 1?D^) and a fourth one as 0 (Other, 
i. e. 22D^) which means it could be treated as an OG as well. Thus, the 
advance classification seemed to be the most promising basis for 
interpretation, making it necessary to re-examine the classification of 
2D ^ for its logico-semantic relationship with the stem. 

Item 2 ; Which of the following factors is the most important 
causitive factor of contempt for stupidity? 

2D^ Compulsory school attendance. ( Sub) 

Probably the best procedure in the analysis of this foil is to 
use a Venn Diagram, as given in Figure 3? page 2J3<> 

Figure 3 shows clearly that "contempt for stupidity" and 
n compulsory school attendance" (Foil 2D^) are disjunctively related. 

For a factor to be "causitive" it must be either conjunctively or 
implicatively related to the other factor. Either conjunction or 
implication is a necessary but not a sufficient condition for causation. 
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FIGURE 3 

VENN DIAGRAM ILLUSTRATING FOIL 2D 

A. Compulsory school attendance. 

B. Contempt for stupidity. 


It would be reasonable to suggest that the student who replaced this 
disjunctive relationahip with a conjunctive relationship, (ignoring the 
fact that either can occur without the other) is substituting one 
category for another. It was upon this basis that this foil (2D ) was 
originally classified as a Sub (Substitution). 

On the other hand, if the student considers the relationship as 
implicative, (i.e. compulsory school attendance can occur without 
contempt for stupidity but not vice vers a) this interpretation could be 
considered an OS (Oversimplification). The thinking in this case 
involves proceeding from an entire set to a subset of that entire set 
as indicated in the use of Arrow #1 in Figure 3* 

If the student begins with the conjunctive subset and extends 
this to include all cases of contempt for stupidity, the thinking 
process would follow the path of Arrow #2 in Figure 3* In such a case, 
(proceeding from a subset to an entire set) the most reasonable 
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interpretation of the thinking is 00 (Cvergeiieralizati-on) . These 
arguments suggest the importance of the way in which an item is 
interpreted with respect to the way in which a foil should be classified. 
As just shown, this foil (2D ) can be reasonably argued to have at least 
three possible classifications depending upon the interpretation placed 
upon the foil by the examinee. 

A decision rule was needed to deal with foils which might 
reasonably be classified in several different categories. Where a 
cluster seemed, in general, to reflect one category of foil, and the 
same interpretation was one of the possible classifications of the 
ambiguous foil, then this classification was assumed to be an appropriate 
interpretation of the ambiguous foil for the particular group on which 
the cluster analysis was conducted. The most important characteristic 
of thus rule was the requirement that the interpretation of a foil 
should probably not be generalized beyond the group of examinees upon 
whom the interpretation was established. 

Hence, since 2 could be an 0G, this wrong answer cluster could 
be interpreted as an 0G cluster for Group A. 

Wrong answer cluster W n was eliminated before logico-semantic 
analysis on the basis of the fact that all of the foils in this cluster 
had a selection ratio of less than o 06. 

For Cluster (see: p. 215) there is no common content. Items 

4, 13 j and 6 comprise all the items in right answer cluster C^ but these 
items represented less than half of the members of W^. Only one of the 
foils has a selection ratio of more than .50, and both 0G and CM have 
two representatives from the advance classification of foils. Hence, the 
interpretation of this cluster could not be established by any of these 
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TABLE 5 T 

INFORMATION FOR THE INTERPRETATION 
OF WRONG ANSWER CLUSTER \I 



Foil 

Content 

Advance 

Classification 

D 

n 


< — 1 
Pi 
P" 

Stupidity 

OG 

O 
1 — 1 

26B 1 

Progress 

0 

1 — 1 
0 


Awareness 

CM 

1 — i 
^0 

6D 2 

Awareness 

Sub 

0 22 

14D„ 

S tup i f i ty , Awarene s s 

OG 

O 

-P 

O 

j 1 

bo 

« 

J\J 

Awareness 

CM 

>07 

29D 2 

Discipline , Progress 

OS 

• 15 


methods. 

In the ensuing discussions a possible common element among the 
foils in a cluster is suggested. The clue which was used in this case 
was the presence of OG, OS. and CM (Common Misconception) in the same 
cluster of which OG and CM were the most frequent. Powell and Isbister 
(1969) found a polarity between OG and 03 on the one hand, and CM on the 
other. This polarity in a factor matrix indicates a significant negative 
correlation between the poles. Since this polarity, at least in part, 
could have been an artifact of the mutually exclusive selection of 
responses within items, it is reasonable to suggest that OS, OG, and CM 
foils may be related. In addition to using the "most frequent" rule, it 
might be possible to reinforce the interpretability of this cluster as 
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CM if most of the remaining foils had CM as one of their possible 
alternative classifications . 

To begin with, foil 2ol)^ as a member of the 0 category can have 
any interpretation which is reasonable for the remainder of the foils • 

A discussion of the others follows: 

Item 4 : The author, in charging that "society teaches contempt 

for stupidity and fear of being regarded as stupid" by 
means of the school, is assuming that: 

4b . The school should not be an enforcing arm for the customs 
of society, (OG) 

This foil is wrong on two counts. In the first place it is a 
conclusion rather than an assumption made by the author 0 Second, it is 
overstated by containing a value judgment which may be unwarranted. 

This second reason led to its OG classification. It is, however, in 
addition, one of the ways of phrasing a very common argument against the 
establishment of parochial schools. In this latter context it could be 
a CM foil as well. However, this argument is shaky at best, and would 
not be likely to extend beyond the context of the group upon which this 
cluster was established. 

Item 29 : On the basis of the foregoing which of the following 

seems to be the most important consideration when 
preparing anecdotal records on progress reports? 

29b. Make no attempts at interpretation since your judgments 
are probably biased. (CM) 

29b 2 Be as brief as possible, giving no information which may 
cloud the central problem 0 (OS) 

Foil 29D was dropped on the basis of its low selection ratio. 
Since in the advance classification two foils in the same class in the 
same item were not entertained, and since the "brief as possible" part 
of 29b 2 is an Oversimplification, foil 29b, was originally classified 


as an OS foil. However 
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•. the phrase 11 which may cloud the central problem 1 ' 
in foil 29D^ i 3 similar in its central idea to "your judgments are 
probably biased." This idea being another way of saying that a person 
should be as objective as possible 0 Students who emphasized this idea 
in their thinking could be responding more to the second part of the 
statement than to the first (OS) part, making CM (Common Misconception) 
a reasonable alternative classification for this foil. Of course, there 
is the problem of the "relevancy of details" which this foil may also 
raise. It is, however, better to put in details which the writer may 
think irrelevant and an independent observer may not than to require the 
independent observer to infer these details from the context because of 
their omissions. This aspect of the discussion leads to another common 
misconception, namely that the simple act of speaking or writing has 
produced a successful communication. 

Item 6 : The purpose of developing a generalized intellectual 

awareness is to: 

6D Stimulate thinking ability within the individual's chosen 
d field. 

The confining of thinking ability to "the individual's chosen 
field" is false. It is a substitution for the phrase "which is not 
contextually bound." Once again, however, the frequency with which this 
misconception is encountered suggests that the foil could be regarded as 
a CM foil as well as a Sub foil. 

Item 14 : Which of the following best describes the probable 

relationship between contempt for stupidity and 
generalized intellectual awareness? 

14D~ Contempt for stupidity should bo reduced and awareness 
should be increased. (OG) 

This foil i s clearly an 00 foil since it adds an unwarranted 
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value judgment to an otherwise correct statement. Should the 
definition of CM foils be extended to incorporate this characteristic? 
In any event, six of the seven foils in this cluster could be assigned 
fairly reasonably to the CM class, making the CM interpretation of the 
entire cluster plausible, if not reasonable. Table 55 on Cluster 
follows. 


TABLE 55 

INFORMATION FOR THE INTERPRETATION 

OF WRONG ANSWER CLUSTER V, 

4 



Foil 

Content 

Advance 

Classification 

D 

n 

30D 3 

Summary of 
all passages 

RT 

o 
1 — 1 



W^ , as a single member cluster, should probably be dropped 
unless a good reason for retaining it can be found. As an approach to 
the problem of classifying this Cluster W^ the fate of other RT foils 
proved helpful 0 Foil 11 v/as originally in this same wrong answer 
cluster (W^) but was dropped because of its low selection ratio. Foils 
13Dq &nd 15IC were both in wrong answer cluster W 7 , and this cluster 
remained uninte.rpretedo The separating factor between 15 L)^, 29L^ , and 
30IL may have been on content lines since each of these are from 

j % 

different reading selections 0 Foil 29L^ also dropped because of i t n 
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low selection ratio. .Foil 13 had similar but not identical background 

to 30D o but was dissimilar to lflh as to content, which was part of the 
3 '3 

classification problem of W ri . The other single member cluster (W-~) 
x 7 u v 10* 

contains foil 29D^ which was originally classified as CM but could also 

be an RTo If the foil in is an RT, the separation is on a content 

basis. The low average total correct scores for these two foils (ll.l 

and 11. 0) may be taken as equivalent except for content; hence, this 

cluster (W, ) was retained as an RT. Table fo on Cluster VL follows. 

4 3 


TABLE 56 


INFORMATION FOR THE INTERPRETATION 

OF WRONG ANSWER CLUSTER W„ 

P 


Foil 

Content 

Advance 

Classi fi cation 

D 

n 

12D 3 

Aggression 

Inv 

olO 

1U) 2 

Aggression 

OS 

.30 

!2D 2 

Aggression 

OG 

.35 

!9P 2 

Progress 

0 

.14 

20.U 3 

Progress 

0 

o3? 


By the rules used thus far, except for the logico-semantic 
interpretation, Cluster should be classified as 0 (Other) since this 
is the most frequently occurring equivalent advance foil class. The 
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unusual element in this cluster, however, is the fact that there are two 
foils from the same item in this cluster. This event violates the 
assumption behind the rule for classification discussed earlier on 
page 153 ? number 5* 

Since the reclassification of any one of foils 12D^, llD p , or 
I 2 D 2 to the same category as either of the others would give that 
classification to four out of the five foils in this cluster, and since 
content, the size of the I) , and the distribution in right answer 
clusters of the corresponding items do not account for this cluster, a 
logico-semantic analysis of the two foils in item 12 would seem 
reasonable 0 

Item 12 : Aggressive behavior in female children is: 

12D^ More unpredictable and expressed differently than in males 
of the same age group. (OG) 

12D^ Less differentiated in expression than in males of the 
same age . ( Inv ) 

Foil 12Iu was classified as Inv (inversion) because it is 
opposite to a true statement. It is not opposite to the actual correct 
answer but to a statement that could have been used as an alternative 
correct answer if the examiner had so chosen. On the other hand, 12D^ 
was classified as OG because the first two words (more unpredictable) 
form an incorrect statement added to a correct statement. However, 
these two words are incorrect by virtue of being opposite to the truth 
(inv) when the restriction M of the same age group” was applied to this 
statement . 

This last property reinforces the Inv (inversion) classification 
of this foil. It may be argued that two Inv foils were possible in 
Item 12 because of the complex logical structure which this item 
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required before it could be answered. (See: the discussion of right 

answer cluster Cg on pages 118 to 120). 

Item 11 : Overt aggression would likely be decreased by: 

11D^ Blocking many inodes of Aggression. (inv) 

IIP Lessening the threat cf punishment. (OS) 

Lessening the threat of punishment, or permissiveness, does not, 
by itself, either increase or decrease overt aggression. It is on these 
grounds that this foil was classified as an OS. Overt aggression will 
be likely to increase if permissiveness develops frustration. * The 
energy of the children must be channelled into alternative directions if 
overt aggression is to decrease in a permissive setting. Thus, it is an 
oversimplification to say that aggression is likely to either increase or 
decrease in a permissive setting. However, lessening the threat of 
punishment, in the absence of alternatives, will probably increase overt 
aggression. This foil could have been classified as an Inv if it were 
not for the fact that 11L^ was already so classified. An alternative 
possible classification for 11D^ is given with the .discussion of Wg 
(see: p. 227). 


Perhaps the reclassification of 11D 0 as Inv is a bit tenuous. 

The other four foils in this cluster are not on such shaky ground, so 

that it is reasonable to reclassify wrong answer cluster W. as Inv. 

5 

Information for the interpretation of Wrong Answer Cluster appears 

o 

in Table 57 on page 222. 

Short of a close logico-semantic analysis of the characteristics 
of the foils in Table 57? there is no clear basis for the interpretation 
of cluster Wg. 


It should be pointed out that the fact that these foils have 


222 


TABLE 57 

INFORMATION FOR THE INTERPRETATION 

OF WRONG ANSWER CLUSTER W, 

o 



Foil 

Content 

Advance 

Classification 

D 

n 


9Di 

Aggression 

Inv 

1 — i 
1 — l 

2 <Di 

Progress 

Sub 

.24 

21 D 

Progress 

0 

.23 

7, 3 

Awareness 

OS 

1 — 1 

vr\ 

26 L 3 

Progress 

I IT 

.13 


formed a cluster has led to the assumption that there must have been a 
common logi co-semantic element in these foils, at least so far as the 
members of Group A were concerned. The argument for the formation of a 
new class of foil (namely: Non Sequitur--NS) which follows should not 

be construed to deny the plausibility of the original classification of 
the foils in this cluster but only as to the inappropriateness of these 
classifications for this particular group of examinees. By this point 
in the study, it had become clear that the foil categories were not 
mutually exclusive, and that it seemed possible to classify at least 
some foils in several different ways (see: 21) , p. 131 ff). Thus, the 

foil interpretations presented here most probably apply only to Group A. 

Item 9: The basic position of the author in writing about 
aggression is that it: 

9D Can be eliminated through the process 
(Inv) 


of socialization. 


22 ; 

On the contrary, Kagan and Moss (!9o2.) assume that aggression is 
one of several innate behavior systems. Being innate makes its elimina- 
tion impossible. However, the socialization process can channel aggres- 
sion away from; its more destructive aspects. A number of classifi- 
cations of this foil are possible. Once again, these classifications 
are dependent upon several possible Icgico-semantic distinctions which 
can be made. Most simply, the foil does not follow from the data, i.e. 
it is a non seoui tur relationship. There was, however, no such classifi- 
cation in the advance classification. This N3 ( non sequi tur ) category 
was eliminated, as indicated above, on the basis that it displayed 
experimental dependencies with CM foils in the Powell and Isbister 
( 1969 ) study. 

Item 25 : The most useful suggestion to help Chester is: 

29B X To give Chester personal warmth, acceptance, and support 
wherever it is appropriate. (Sub) 

As the right answer indicates, Chester needs "concrete help in 
getting started on specific tasks.” In other words his problems would 
seem, from the progress report, more developmental than emotional. The 
procedure given in 25 D^ substitutes a treatment procedure designed to 
deal with emotional problems for a procedure designed for developmental 
problems. By itself, the use of 25B-^ is simply inappropriate. Re- 
classifying 2913-^ into a new HS category would seem to be reasonable. 

Item 21 : Chester is growing very slowly and really quite 

immature for his grade. Everyone expects too much of 
him. 


21 This hypothesis is refuted by the facts. (0) 

On the contrary, the only information directly available about 
Chester* s physical development is the number of days he has been absent 
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or tardy® This limited irifcrmatioij is insufficient concerning the 
physical areas of development given in Item 21 to form any conclusions. 
He shows some indications of specific academic: immaturity, and perhaps 
some social immaturity, but this is all. As to v/hether or not the 
expectations made of him are unreasonable, it cannot be decided from the 
information given* The only expectation statement made about his w ork 
is "Is not ready for division," which does not sound like a statement of 
overexpectation. 

Of course, 21D 0 could have been reclassified without this 
analysis because of its 0 (Other) advance classification. However, 
its relationship to the correct answer supported the use of an NS 
category for this foil* 21D Q was dropped for underselection, and 21D^ 
formed part of one of the two unnamed new categories to emerge from this 


study. 


(^ 13 ). 


information also implies the reasonableness of the 


formation of a new NS category for \irong answer cluster Wg. 

Item 7 : Of the following the best example of generalized 

intellectual skill is: 

7I> 3 Applying abstract principles to new situations. (OS) 

Comparing 73)^ with the "correct" answer "The widely applicable 
technique of logic" showed this foil to be without question an OS foil, 
since it is a. true statement of narrower generality than the correct 
answer. As a true statement it cannot be classified as an NS, which 
leaves the interpretation of this cluster based on this item in some 
doubt. Any alternative interpretations such as involving its large 
selection ratio (.51) or suggesting a tenuous link between NS ( Non 
Sequitur ) and OS (Oversimplification) would be premature at this stage. 
Cross validation data helps to clarify this cluster somewhat* The fact 
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that this foil moves to wrong answer cluster in Group B which is most 

prominantly composed of members from V/^ (the wrong; answer cluster from 

Group A which proved to be unclassif table) relieves the problem of 

interpreting this cluster somewhat, but does not solve it. Foils 9B-^ 

and 25 D^ moved together, as did 21D^ and ?D^, while 26D^ migrated by 

itself to a new cluster. This cluster held together better than any other 

in cross validation with the exception of V/^. 

Item 2 6 : If additional information on Chester is desired and 

none of the following has been attempted, which one 
would provide the greatest amount of immediately 
useful information: 

26D~ A reouest for the assistance of a guidance counselor. 

J (irrj 

The D (selection ratio) on this foil is .13* It might be 
n v 

interesting to know who made these selections. Most of the examinees 
in this group were practicing teachers and they would know from experi- 
ence that the usual information from the guidance counselor would merely 
reinforce what they already knew and not add much further useful 
information. In the absence of an NS category, this foil is clearly an 
Irr. 


Of the five foils in this cluster, four could have been 
classified quite reasonably as NS foils. The fifth one (7Lu) could not 
have been so classified meaning that this cluster could not be unambig- 
uously classified. However, the best interpretation for this cluster 
within Group A is to establish a new class of foil, namely: Non 

Sequitur (NS) : This type of foil is a wrong answer foil by virtue of 
the fact that it simply does not follow from the given 
information. As a rule some part of the foil stands in 
direct contradiction to the logical stricture or 
connotative moaning of the background information (or 
part thereof) required to answer the question & 


Wrong answer cluster W^ waj classified as containing NS foils, 
at least so far as Group A was concerned. Table 58 giving information 
for the interpretation of wrong answer Cluster W 0 follows. 


TABLE 58 

INFORMATION FOR THE INTERPRETATION 
OF WRONG ANSWER CLUSTER W 7 


Foil 

Content 

Advance 

Cl a s 3 i f i cation 

I) 

n 

10D 1 

Aggression 

CM 

. 22 

19 D 3 

Progress 

0 

.47 

4d 3 

Stupidity 

Irr 

.72 

13I' 3 

3 tup i d i t y , A wa r ene s s 
Aggressi on 

RT 

.24 

12D 

Aggression 

CM 

.33 

15 D 3 

Discipline 

RT 

.88 

18D ? 

Discipline 

Irr 

\0 

00 

O 

26D 2 

Progress 

OS 

.34 

8D 3 

Aggression 

Inv 

.12 


The information in Table 58 provides no clear basis upon which 
to interpret wrong answer cluster W . It contains two CM's, xwo RT's, 
and two Irr's. In general, the lower D^ 1 s relate to the selection on 
aggression (2 6D is the exception). 


No detailed discussion will be given for this group. One 
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illustration is sufficient. Foils 10 and 1211^ could conceivably be 
reclassified as RT foils on the basis of log: co-semantic analysis, as 
could 19b^ by the M 0" rule* However, such a reclassification is un- 
reasonable for foils 41b , 18D , 26l) , and 81V. Similar findings 

J 

occurred for other pairs in this wrong answer cluster. 

The inability to interpret this cluster led to its being dropped 
from further relevant analysis. This was somewhat unfortunate since 
several of these foils have a high selection ratio meaning that a fair 
amount of information was lost by this decision. Nonetheless, it is 
reasonable to argue that if a wrong answer cluster cannot be given an 
adequate label, it should not be used. Table 59 giving information for 
the interpretation of wrong answer cluster Wg follows. 


TABLE 59 

INFORMATION FOR THE INTERPRETATION 
OF WRONG ANSWER CLUSTER W Q 






Foil 

Content 

Advance 

Class 1 f i cat i on 

3 

n 


11 D 

Aggression 

Inv 

.20 

r\3 

00 

fc* 

oJ 1 

S tup i d i ty , Pr ogre s s 

IA 

o06 

14D„ 

Stup i d i ty , A war ene s s 

Sub 

O 
i — l 


Table 59 shows that Cluster V Q also seems to be ambiguous. 
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Item II: Overt aggression would likely 'be decreased by: 

11D^ Blocking of many modes of aggression. (Inv) 

There are several ways in which this foil can be interpreted. 

The blocking of modes of aggression would not reduce overt aggression 
except in those specific areas where the blocking occurred. Overt 
aggression would increase in areas where the blocking was absent or less 
effective, or more socially acceptable to the peer group. If the 
blocking increased the frustration level, the absolute incidence or overt 
aggression would also increase. For this latter reason, (i.e. since 
blocking would probably have the opposite to the desired effect) this 
foil was classified as an Inv. However, to arrive at this conclusion as 
being correct the examinee must make the invalid assumption that attempts 
to regulate overt behavior also regulate innate drives. This foil could, 
therefore, be classified as an IA foil. This decision gave the cluster 
a majority of IA foils. If IA was also a reasonable alternative for 
l4l) 0 , then IA would be a reasonable classification for this cluster so 
far as Group A was concerned. 

Item 14 : Which of the following best describes the probable 

relationship between contempt for stupidity and 
generalized intellectual awareness? 

I 4 B 2 Either will increase with an increase in the other. (Sub) 

Foil 14D 2 was one of the foils about which the raters shewed 

considerable disagreement. It was classified as Sub because part of the 

relationship was wrong, and it did not seem to relate to the other Inv 

foils in the pilot study. This relationship could also be an Inv, 

because a direct relationship is logically opposite to an inverse 

relationship. Powell and Isbister ( 1969 ) encountered the same problem 

in logical relations (as compared to logical fallacies) type foils. On 


the other hand, this foil could wrongly bo considered correct if the 

examinee forms an invalid assumption relating "critical’ 1 thinking with 
contempt for stupidity by defining stupidity in terms of uncritical 
thinkingo 

Other possible classifications were given to this foil with 
similarly tangled arguments, .justifying each rater’s conclusions. 

Two of these foils can be reasonably regarded as IA foils, the 
third could well be stretching the point. In any case, a reasonable 
over-all classification for this wrong answer cluster would seem to be 
IA. Table 60 giving information for the interpretation of wrong answer 
cluster follows. 


TABLE 60 

INFORMATION FOR THE INTERPRETATION 
OF WRONG ANSWER CLUSTER V 


Foil 


Content 


Advance 

Classification 


I) 

n 


3D 3 

Stupidity 

28D ] 

Progress 

iod 2 

Aggression 

17D 3 

Discipline 

27D ? 

Progress 

5D 3 

Awareness 


08 

.41 

Irr 

.25 

Inv 

.06 

Sub 

ON 

O 

0 

. 06 


OS 


12 
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In Table 60 the two OS’s and the 0 foil account for half of the 

foils in Cluster W Q making an OS classification of this cluster within 

> 

the rules based on the advance classification. It would be better if the 
other three could be alternatively classified as OSo 

Item 28 : In Chester's progress report, which one of the 

following is the most important factor contributing to 
his difficulty with school achievement? 

28h^ His ability to develop a generalized intellectual 

ci/ \ » X ^ Xl O O o • y) X X X y 

This foil was classified as Irr on the basis that in Grade 6 most 
children are still too young to have progressed very far into '‘Formal 
Operations" which form the basis of generalized intellectual awareness. 
This classification assumes that the examinees know this information 
about development which is an unreasonable assumption. In the absence 
of this information, Chester's problems also involve decoding skills in 
reading and to a lesser extent in arithmetic. Hence his problems involve 
mere than this foil suggests, and the foil might, for this reason, he 
reclassified as an OS foil. 

Item 10 ; With which of the following statements concerning 

aggression would the author be most likely to agree? 

lOD^ Aggression generally interferes with the attainment of 
educational goals. (inv) 

This foil was classified as an Inv because the best answer was 
"Aggression is potentially useful for educational pur-poses 0 " Superfi- 
cially lOD^ wou ^-d seem to be opposite to the right answer D On the other 
hand, aggression can interfere with the educational process. Aggression 
expressed in the form of competition may he a useful form of intrinsic 
motivation. The term "generally" in the foil overstates the case for 
the negative aspects of aggression whereas the foil itself understates 


231 


the total picture. This foil might be classified as either OS or OG 
depending on how the examinee locks at the item. 

Item 17 : From the above passage we can infer that Doris' 

leadership of the group was : 

17D 3 Laissez-faire . ( Sub ) 

The reason for classifying this foil as Sub was that Doris' 
attempts to coerce were ineffective, sc she let the teacher take oveiu 
As a result her later leadership was laissez-faire but only under the 
arbitrary intervention of the teacher. As pointed out in the original 
discussion of this item, even this argument is stretching the point. 

(See: pp. 172,173)* There is no similar plausible argument which might 

make this foil an 0S o 

Five of the six foils in this cluster can be included in the OS 
categories hence the advance classification of this wrong answer cluster 
is retained. Information for the interpretation of wrong answer cluster 
follows in Table 6l. 


TABLE 61 

INFORMATION FOR THE INTERPRETATION 

OF WRONG ANSWER CLUSTER W, . 

iu 




Advance 

D 

n 

Foil 

Content 

Classification 

29 \ 

Discipline , progress 

CM 

CO 

o 


given in Table 61 is the 


Cluster W_ as 
10 


second of two single 


member clusters. It contains only 291^ which was originally classified 
as a CM for the reasons already given (see: pp. 180,181)* However, the 
fact that it did not occur in a common cluster with the other CM foils 
raises some doubts over this classification within the confines of 
Group A. It is possible, however, to classify this foil into other 


categories . 

Item 29 : On the basis of the foregoing which of the following 

seems to be the most important consideration when 
preparing anecdotal records or progress reports? 

29D Make no attempts at interpretation since your judgments 
are probably biased. (CM) 

The suggestion in this foil that all interpretations are 
sufficiently biased so as to be of little value has the effect of re- 
defining the term "interpretation" to mean "biased interpretation." 

In this case, it would be necessary to assume that the other 
single member group (W^) which is also an RT (Redefinition of Terms) 
split from this one along content lines. The obvious vocabulary- content 
linkage in both of these foils makes this conclusion reasonable. Foil 
29D ][ was reclassified, therefore, as RT (Redefinition of Terms) making 
Cluster \J an RT cluster. The apparent content binding of some mis- 
reading type foils would seem reasonable, since there seems to be a 
parallel group of foil levels to the right answer levels in Bloom's 
Taxonomy . Bloom's description of the levels in the Taxonomy suggest a 
steady progression av/ay from context as the level of the categories 
increase. For this reason, it seemed reasonable to combine W 1 ^ with 
as an RT cluster in subsequent analysis, rather than to discard both 


clusters because of their small size. 
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Table 6.2 gives information for the 


i n t c rp vo t a t i on 


of wrong 


answer cluster W 1 <. 


TABLE 62 

INFORMATION FOR THE INTERPRETATION 


OF WRONG ANSWER CLUSTER W. 


11 



Foil 

Content 

Advance 

Classification 

i) 

n 


18D 1 

Discipline 

Sub 

.11 

?D 2 

Awareness 

WW 

.31 

6D 3 

Awareness 

Irr 

.23 

16D 3 

Discipline 

CM 

.14 

2D 2 

Stupidity 

no 

CO 

.09 

10L 3 

Aggression 

WW 

°55 


In Table 62 the most frequent foil category in Cluster W ^ was 
WW (Word-Word Link) which serves by the decision rules to identify this 
cluster. 


A logico-semantic analysis of the other foils might reinforce 
this classification. 

Item 18 : From the description of the incident, we can conclude 

that the teacher’s handling of the incident was: 

18D^ Good: she intervened to prevent a serious conflict from 

coni. 1 nu i ng . ( Sub ) 

The reading selection states: ’’There seemed to be confusion in 


this group so I decided to investigate. M 


The similarity between this 
statement and the phrase "intervened to prevent” in the foil is self- 
evident. WW would seem to be a reasonable alternative classification 
for this foil. 

Item 6 : The purpose of developing a generalized intellectual 

awareness is to: 

6lL Give the individual an ever-widening view of his world, 
(irr) 

The similarity between the phrase "ever-widening view" in this 
foil and the phrase "free-ranging understanding" in the reading 
selection warranted the use of the alternative class of WW for this foil 

Item 2 : Which of the following is the most important causative 

factor of contempt for stupidity. 

2D 9 Compulsory written examinations. (OS) 

The phrase "compulsory written examinations" occurs in both this 
foil and the reading selection. 

Item 16 : If the teacher hau written "Doesn’t work well with 

others," as an anecdotal record for the above incident 
this would have been: 

l6Do Worse: Teachers are failing in their obligations in not 

supplying complete information. (CM) 

There is no similar connection in foil 16D^ between the stem or 
the reading selections to the ones presented above. The above discus- 
sion, nonetheless, in general supports the retention of WW as a 
reasonable interpretation for this wrong ansv/er cluster. 

Since, as shown in Table 63 (see: p. 235) both of these foils 
(19D^ and 24D^) are classified as 0 and both come from the same content, 
there may be some doubt about the interpretation of this cluster 0 

One obvious course of action with this foil cluster would be to 
drop it from further analysis*, 0 n the other hand, the foil classes in 
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TABLE 6 3 

INFORMATION FOR THE 1NTEPJ? RETATT ON 
OF WRONG ANSWER CLUSTER V 



Foil 

Content 

Advance 

Cla s s i f i ca t ion 

D 

n 

19D, 

m 2 

Progress 

Progress 

0 

0 

.09 

.O' 7 



the Guidelines made no pretense at being exhaustive. Since the 
experience with other foil clusters has been that logico-semantic 
snaiysis has often revealed a possible common base for interpreting 
foils within a particular cluster, such an analysis for this cluster may 
assist in the illumination as to how the list of Guidelines might be 
extended. For this reason, these foils were also analysed. 

To begin with, the fact that they stood separate from any of the 
interpreted categories suggests that these two foils may form a foil 
class, the basis of which has not yet been determined, but possibly 
related to the special format of items 19 to 2^1 inclusive 0 

Items 19 to 24 inclusive used a classification protocol designed 
to get the examinees to treat the statements represented by these items 
as hypotheses to be tested on the basis of the information in the reading 
selection from Prescott (1957) concerning Chester f s Progress Report. The 
categories were: 


1. supported. 
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2. implied. 

3. refuted. 

4. insuff i eient evidence 0 

In these categories there is a hierarchy of inferential support 
from insufficient evidence, to implied, to support 0 

The "correct" answer from item 19 is "implied" and the 
corresponding answer given in 19E^ was Supported . " Similarly, the 
tl correct 11 answer for item 24 was "insuff icien t evidence” and the 
corresponding answer in this cluster was "implied." Each answer given 
in these foils was one step higher in the hierarchy than the "correct" 
answer. This "overstatement" was not the same as "overgeneralization" 
as defined in this study 0 Whether or not such a relationship is 
exclusive to this type of question remains undetermined. Rather than 
premature naming, the 0 (Other) classification of this cluster v/as 
retained. Since another cluster was interpreted as 0 (i.e. W n ^), a 
subscript was applied for purposes of distinguishing between these two 
clusters. 

Once again, Cluster as given in Table 64 (see: p* 156) is 

an 0 classification based on the advance foil classification. The 
logico-semantic analysis of this cluster proved to be very interesting. 

A slight change of format will be used in this discussion with all 
three items being presented before the discussion of them as a cluster. 

Item 21 : Chester is growing very slowly and is quite immature 

for his grade. Everyone expects too much of him. 

21D^ The hypothesis is supported by the facts 0 (0) 

Item 28 : In Chester's progress report, which one of the 

following is the most important factor contributing to 
his difficulty with school achievement. 

28D^ His weakness in reading which is affecting all areas of 


Item 23 : Cheater's reading deficiency has not yet begun to 

affect seriously his performance in other areas. 

23l>2 The hypothesis is refuted by the facts. (0) 


TABLE 64 

INFORMATION FOR THE INTERPRETATION 

OF WRONG ANSWER CLUSTER W n „ 

1 J 



Foil 

Content 

Advance 
Classif i cation 

D 

n 


21D 1 

Progress 

o 

,07 

28D 2 

Progress 

OG 

.32 

23D. 

> 

progress 

0 

• 1? 


As it happens, the hypothesis in item 23 is supported rather 
than refuted by the facts. Notice, however, that this same contrary- 
to-fact conclusion is stated in foil an ^ implied in the response 

to item 21. The positive statement of this false conclusion was most 
frequently selected, (i.e. 28D^ for which the B was o32) 0 The negative 
statement (21D^) was less frequently selected (D^ = 0 17) and the implied 
statement (21D^) least frequent (D^ = <>07) • This would seem to be 
reasonable o Also, this cluster holds together better than any other in 
cross validation (two of the three form a new three-member cluster). 
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It would seen i, then, that this cluster is content bound, not so 
much on the basis of a common reading selection, but rather upon a single 
common contrary r- to-. fact conclusion formed by some of the examinees. 

The 0 (Other) designation was therefore, retained making this 
wrong answer cluster 0^° Table 65 giving information on Cluster 
follows . 


TABLE 65 

INFORMATION FOR THE INTERPRETATION 
OF WRONG ANSWER CLUSTER V ^ 


Foil 

Content 

Advance 

Clan si f i cation 

D 

n 

3D 2 

Stupidity 

Irr 

o23 

25D ? 

Progress 

Irr 

• 19 

9D 2 

Aggression 

Sub 

.17 

27D 2 

Progress 

Irr 

.09 


Wrong answer Cluster ^ 
four cases. Foil 9If cannot be 
not a true statement. However, 
was retained on the basis of the 
(see: p. 23?) » Table 66 giving 

wrong answer cluster ^ follows 


contained Irr foils in three out of 
reclassified as an Irr foil since it i 
the Irr classification of this cluster 
decision rules already discussed 
information for the interpretation of 
(sees p. 2'39)„ 


s 


TABLE 66 


INFORMATION FOR THE INTERPRETATION 

OF WRONG ANSWER CLUSTER W. c 

J-5 


Foil 


30D 2 

24Dj 


Content 

Advance 

Classification 

D 

n 

Summary of all 



Sele et i on s 

Tr 

ol5 

Progress 

0 

„06 


Wrong answer cluster W^ r , as given in Table 66, could not be 
interpreted without logic-o- semantic analysis because it did not contain 
a most frequent category by the advance foil classification,, A rea- 
sonable approach would be to investigate the possibility that 24D^ might 
also be classified as Tr. 


I tem 24 : Chester’s mother has kept after him about his reading 

until he hates it. 

24D^ The hypothesis is refuted by the facts. 

Several facts can be derived from Chester's ’’Progress Report” 
which have a bearing on Item 24. The three most important are: 

1. ’’Does not enjoy reading.” 

2. ’’Has a wide speaking vocabulary.” 

3. ’’Enjoys spelling.” 

From these three facts two conclusions can be drawn: 
l) Chester’s background must be fairly verbal, hence his 

reading deficiency is probably not the direct product of a 
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background d i sadvan tage o 

2) Since he likes spelling and he "is better able to find facts 
than to interpret facts," his reading problem would seem to 
be a decoding problem® 

In addition, the report gives no direct evidence about Chester’s 
home background® Hence the best answer for this question is "insuffi- 
cient evidence o" In order to have the proposition presented in Item 24 
refuted by the facts more would have to be known about the probable 
sources of the decoding problems and the side speaking vocabulary. Only 
if these two considerations were emphasized beyond reason could the 
refutation be acceptable® This conclusion might be expected to have 
arisen, then, from a reordering of the emphasis given to parts of the 
reading selection vis £ vis other parts of the selection® The classifi- 
cation of 241) as Tr would seem to bo appropriate in this case, giving 


the entire cluster this 


same interpretati on 


